Sep 10 19:08:58.890834 ip-10-0-141-45 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 19:08:58.890849 ip-10-0-141-45 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 19:08:58.890859 ip-10-0-141-45 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 19:08:58.891070 ip-10-0-141-45 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 19:09:09.108738 ip-10-0-141-45 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 19:09:09.108756 ip-10-0-141-45 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot aee666496b7d43ea9178e65f7f474b50 --
Sep 10 19:11:03.144558 ip-10-0-141-45 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 19:11:03.563745 ip-10-0-141-45 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 19:11:03.563745 ip-10-0-141-45 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 19:11:03.563745 ip-10-0-141-45 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 19:11:03.563745 ip-10-0-141-45 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 19:11:03.563745 ip-10-0-141-45 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 19:11:03.565520 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.565434    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 19:11:03.569881 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569865    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:11:03.569881 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569880    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:11:03.569881 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569884    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569887    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569890    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569893    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569896    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569899    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569904    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569908    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569912    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569914    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569917    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569921    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569924    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569927    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569930    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569933    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569935    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569938    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569940    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:11:03.569977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569943    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569945    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569948    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569950    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569954    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569956    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569958    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569961    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569963    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569965    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569968    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569973    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569975    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569979    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569981    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569984    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569987    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569989    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569992    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569994    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:11:03.570478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569997    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.569999    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570002    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570004    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570006    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570009    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570011    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570014    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570016    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570019    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570021    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570025    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570029    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570032    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570035    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570038    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570040    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570043    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570045    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:11:03.570948 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570048    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570050    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570053    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570055    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570058    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570060    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570066    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570069    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570072    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570075    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570079    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570081    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570084    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570086    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570089    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570092    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570094    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570097    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570099    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:11:03.571468 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570102    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570104    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570107    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570109    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570112    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570114    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570117    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570119    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570122    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570124    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570127    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570129    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570132    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570135    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570138    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570140    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570142    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570145    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570148    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570151    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:11:03.572267 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570835    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570842    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570844    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570847    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570850    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570853    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570856    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570858    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570861    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570863    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570865    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570868    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570870    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570873    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570875    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570878    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570880    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570883    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570885    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570888    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:11:03.573005 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570891    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570893    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570896    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570898    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570900    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570903    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570906    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570909    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570911    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570913    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570916    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570918    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570921    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570924    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570928    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570931    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570934    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570937    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570940    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:11:03.573703 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570942    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570944    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570947    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570949    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570952    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570954    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570957    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570959    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570962    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570965    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570967    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570970    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570972    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570976    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570979    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570982    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570985    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570987    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570990    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570992    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:11:03.574542 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570995    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.570998    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571000    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571003    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571005    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571007    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571010    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571013    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571015    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571018    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571020    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571022    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571025    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571027    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571030    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571032    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571036    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571040    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571044    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:11:03.575431 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571046    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571049    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571051    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571054    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571056    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571058    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571061    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571063    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571066    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571069    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571071    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571074    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571076    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571078    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571082    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571085    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571089    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571091    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571094    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571096    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:11:03.576139 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.571099    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573734    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573750    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573755    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573761    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573767    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573771    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573777    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573782    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573786    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573791    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573795    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573800    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573804    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573809    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573813    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573817    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573823    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573827    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:11:03.576689 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573832    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573836    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573841    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573846    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573850    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573855    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573859    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573863    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573867    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573871    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573875    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573880    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573885    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573890    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573894    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573899    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573903    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573907    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573911    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573916    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:11:03.577223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573920    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573924    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573928    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573936    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573942    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573947    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573953    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573961    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573965    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573970    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573975    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573979    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573984    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573988    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573992    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.573996    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574001    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574004    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574008    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:11:03.577892 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574013    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574017    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574024    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574027    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574031    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574036    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574040    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574045    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574049    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574054    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574058    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574062    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574068    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574074    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574078    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574082    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574086    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574091    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574095    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574099    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:11:03.578387 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574103    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574107    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574111    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574115    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574119    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574123    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574129    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574134    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574138    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574142    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574146    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574150    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574154    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574158    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574162    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574165    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574169    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574173    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574178    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574181    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:11:03.578974 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574186    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.574190    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574313    2573 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574368    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574378    2573 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574385    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574392    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574397    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574403    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574410    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574415    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574420    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574425    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574431    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574435    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574440    2573 flags.go:64] FLAG: --cgroup-root=""
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574445    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574450    2573 flags.go:64] FLAG: --client-ca-file=""
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574454    2573 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574461    2573 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574469    2573 flags.go:64] FLAG: --cluster-domain=""
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574473    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574478    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574483    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 19:11:03.579478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574489    2573 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574496    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574501    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574506    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574511    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574516    2573 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574521    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574525    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574531    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574536    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574541    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574546    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574550    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574554    2573 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574559    2573 flags.go:64] FLAG: --enable-server="true"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574563    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574570    2573 flags.go:64] FLAG: --event-burst="100"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574575    2573 flags.go:64] FLAG: --event-qps="50"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574580    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574585    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574590    2573 flags.go:64] FLAG: --eviction-hard=""
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574595    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574600    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574605    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574609    2573 flags.go:64] FLAG: --eviction-soft=""
Sep 10 19:11:03.580070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574614    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574618    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574623    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574628    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574633    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574637    2573 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574642    2573 flags.go:64] FLAG: --feature-gates=""
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574651    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574655    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574660    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574665    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574670    2573 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574675    2573 flags.go:64] FLAG: --help="false"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574680    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-141-45.ec2.internal"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574685    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574690    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574695    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574700    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574706    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574710    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574715    2573 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574720    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574724    2573 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 19:11:03.580729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574729    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574734    2573 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574738    2573 flags.go:64] FLAG: --kube-reserved=""
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574743    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574748    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574753    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574758    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574763    2573 flags.go:64] FLAG: --lock-file=""
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574767    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574771    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574776    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574785    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574790    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574795    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574800    2573 flags.go:64] FLAG: --logging-format="text"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574805    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574812    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574816    2573 flags.go:64] FLAG: --manifest-url=""
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574821    2573 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574828    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574833    2573 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574840    2573 flags.go:64] FLAG: --max-pods="110"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574844    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574849    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574854    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 19:11:03.581351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574860    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574864    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574869    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574873    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574884    2573 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574889    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574894    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574899    2573 flags.go:64] FLAG: --pod-cidr=""
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574903    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574912    2573 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574917    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574922    2573 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574926    2573 flags.go:64] FLAG: --port="10250"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574931    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574935    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-002e8ea61a93720f9"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574940    2573 flags.go:64] FLAG: --qos-reserved=""
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574945    2573 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574949    2573 flags.go:64] FLAG: --register-node="true"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574954    2573 flags.go:64] FLAG: --register-with-taints=""
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574960    2573 flags.go:64] FLAG: --registry-burst="10"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574965    2573 flags.go:64] FLAG: --registry-qps="5"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574970    2573 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574975    2573 flags.go:64] FLAG: --reserved-memory=""
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574981    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574987    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 19:11:03.582055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574992    2573 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.574997    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575001    2573 flags.go:64] FLAG: --runonce="false"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575006    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575011    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575015    2573 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575020    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575025    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575029    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575034    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575039    2573 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575044    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575049    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575053    2573 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575058    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575063    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575067    2573 flags.go:64] FLAG: --system-cgroups=""
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575072    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575081    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575085    2573 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575090    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575096    2573 flags.go:64] FLAG: --tls-min-version=""
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575101    2573 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575105    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575110    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 19:11:03.582700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575114    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575119    2573 flags.go:64] FLAG: --v="2"
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575125    2573 flags.go:64] FLAG: --version="false"
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575132    2573 flags.go:64] FLAG: --vmodule=""
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575138    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575144    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575351    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575359    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575364    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575369    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575373    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575378    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575382    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575388    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575392    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575396    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575401    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575405    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575409    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575414    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:11:03.583392 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575418    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575424    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575430    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575434    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575438    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575442    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575446    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575449    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575453    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575457    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575461    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575465    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575472    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575477    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575482    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575487    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575491    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575496    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575502    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:11:03.583886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575507    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575511    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575515    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575519    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575523    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575527    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575533    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575537    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575541    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575545    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575549    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575553    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575557    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575562    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575566    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575570    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575574    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575578    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575582    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575586    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:11:03.584409 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575590    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575594    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575598    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575602    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575606    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575610    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575614    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575621    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575627    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575631    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575635    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575642    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575647    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575650    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575654    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575658    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575663    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575667    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575672    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:11:03.584910 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575676    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575680    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575684    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575689    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575694    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575698    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575702    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575707    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575711    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575715    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575719    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575722    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575726    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575730    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575734    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575738    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575742    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575746    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575750    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575754    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:11:03.585403 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575758    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575762    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575766    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575770    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575776    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575781    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.575785    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.575793    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.584655    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.584673    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.585461    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.585473    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585538    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585544    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:11:03.585884 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585549    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585553    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585557    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585560    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585563    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585566    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585569    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585571    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585574    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585577    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585579    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585582    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585584    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585587    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585589    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585592    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585595    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585598    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585600    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585603    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:11:03.586227 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585605    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585608    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585610    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585613    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585615    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585618    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585620    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585624    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585627    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585629    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585632    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585635    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585637    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585640    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585642    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585645    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585648    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585652    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585655    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585658    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:11:03.586735 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585661    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585663    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585666    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585669    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585671    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585674    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585676    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585679    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585682    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585684    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585687    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585689    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585692    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585694    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585697    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585699    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585702    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585704    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585707    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:11:03.587223 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585710    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585713    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585715    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585718    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585720    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585723    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585725    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585727    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585730    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585732    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585735    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585737    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585740    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585743    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585745    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585748    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585750    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585752    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585755    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585758    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:11:03.587730 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585760    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585763    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585766    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585770    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585773    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585775    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585778    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585780    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585782    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585785    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585787    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585789    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585792    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585795    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585798    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585800    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585802    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:11:03.588217 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585805    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.585810    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585922    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585927    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585930    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585933    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585935    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585938    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585941    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585944    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585946    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585949    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585952    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585954    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585958    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:11:03.588661 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585960    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585963    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585965    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585968    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585970    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585973    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585975    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585978    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585980    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585983    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585985    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585987    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585990    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585993    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.585996    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586000    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586004    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586007    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586010    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:11:03.589036 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586012    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586015    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586017    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586020    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586022    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586024    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586027    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586029    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586031    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586034    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586036    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586039    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586041    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586044    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586047    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586049    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586053    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586056    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586059    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:11:03.589510 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586062    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586064    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586067    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586070    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586072    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586074    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586077    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586080    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586082    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586084    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586087    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586090    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586092    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586094    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586097    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586099    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586102    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586104    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586106    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586109    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:11:03.589977 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586111    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586115    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586118    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586121    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586123    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586126    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586129    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586132    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586135    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586137    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586140    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586142    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586145    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586148    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586150    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586153    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586155    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586158    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586160    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:11:03.590474 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586163    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586166    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586168    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586170    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586173    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586176    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586178    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586180    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:03.586183    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.586188    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.586329    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.588606    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.589953    2573 server.go:1019] "Starting client certificate rotation"
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.590052    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 19:11:03.590928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.590091    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 19:11:03.593285 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.593270    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 19:11:03.594448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.594427    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 19:11:03.607726 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.607704    2573 log.go:25] "Validated CRI v1 runtime API"
Sep 10 19:11:03.611053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.611029    2573 log.go:25] "Validated CRI v1 image API"
Sep 10 19:11:03.611916 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.611900    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 19:11:03.615503 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.615483    2573 fs.go:135] Filesystem UUIDs: map[0a20a6c3-8d38-459e-a8b4-6fb6e3c2489b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f41befc5-0703-4784-8b7f-ec8cd3298536:/dev/nvme0n1p4]
Sep 10 19:11:03.615575 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.615502    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 19:11:03.621043 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.620917    2573 manager.go:216] Machine: {Timestamp:2026-09-10 19:11:03.619091788 +0000 UTC m=+0.349787690 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3085854 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec294bb0103a24bc09e882b5f88e9976 SystemUUID:ec294bb0-103a-24bc-09e8-82b5f88e9976 BootID:aee66649-6b7d-43ea-9178-e65f7f474b50 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4d:41:19:f1:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4d:41:19:f1:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:7d:36:ab:d4:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 19:11:03.621043 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.621034    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 19:11:03.621172 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.621111    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 19:11:03.624525 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.624500    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 19:11:03.624691 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.624528    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 19:11:03.625447 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.625435    2573 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 19:11:03.625509 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.625452    2573 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 19:11:03.625509 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.625463    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 19:11:03.626198 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.626186    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 19:11:03.626608 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.626584    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 19:11:03.627478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.627463    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 19:11:03.631794 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.631780    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 19:11:03.631934 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.631923    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 19:11:03.631989 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.631962    2573 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 19:11:03.631989 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.631975    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 19:11:03.632083 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.631991    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 19:11:03.632083 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.632005    2573 kubelet.go:398] "Adding apiserver pod source"
Sep 10 19:11:03.632083 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.632017    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 19:11:03.632453 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.632440    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 19:11:03.632546 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.632465    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 19:11:03.634679 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.634663    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 19:11:03.637310 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.637295    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 19:11:03.637365 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.637326    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 19:11:03.637487 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.637477    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 19:11:03.637527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.637492    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 19:11:03.637527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.637498    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 19:11:03.637527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.637505    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 19:11:03.637527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.637513    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 19:11:03.637527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.637518    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 19:11:03.638018 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.638010    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 19:11:03.638047 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.638018    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 19:11:03.638047 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.638025    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 19:11:03.638047 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.638031    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 19:11:03.638047 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.638040    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 19:11:03.638047 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.638047    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 19:11:03.638861 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.638851    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 19:11:03.638861 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.638861    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 19:11:03.645047 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.645026    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 19:11:03.645134 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.645082    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 19:11:03.645134 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.645084    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 19:11:03.648267 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.648235    2573 server.go:1268] "Started kubelet"
Sep 10 19:11:03.648333 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.648307    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 19:11:03.648425 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.648372    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 19:11:03.648471 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.648452    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 19:11:03.649292 ip-10-0-141-45 systemd[1]: Started Kubernetes Kubelet.
Sep 10 19:11:03.652192 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.652175    2573 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 19:11:03.653239 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.653213    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 19:11:03.654629 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.654611    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 19:11:03.654964 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.654069    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-45.ec2.internal.18d40b6119f5cc4e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-45.ec2.internal,UID:ip-10-0-141-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-45.ec2.internal,},FirstTimestamp:2026-09-10 19:11:03.648201806 +0000 UTC m=+0.378897707,LastTimestamp:2026-09-10 19:11:03.648201806 +0000 UTC m=+0.378897707,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-45.ec2.internal,}"
Sep 10 19:11:03.657160 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.657135    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 19:11:03.657723 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.657705    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 19:11:03.658006 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.657984    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:03.658387 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658366    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 19:11:03.658387 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658388    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 19:11:03.658534 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658368    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 19:11:03.658534 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658469    2573 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 19:11:03.658534 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658477    2573 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 19:11:03.658633 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658551    2573 factory.go:55] Registering systemd factory
Sep 10 19:11:03.658633 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658573    2573 factory.go:223] Registration of the systemd container factory successfully
Sep 10 19:11:03.658900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658885    2573 factory.go:153] Registering CRI-O factory
Sep 10 19:11:03.658900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658900    2573 factory.go:223] Registration of the crio container factory successfully
Sep 10 19:11:03.658997 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658945    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 19:11:03.658997 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658959    2573 factory.go:103] Registering Raw factory
Sep 10 19:11:03.658997 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.658970    2573 manager.go:1196] Started watching for new ooms in manager
Sep 10 19:11:03.659363 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.659349    2573 manager.go:349] Starting recovery of all containers
Sep 10 19:11:03.666577 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.666457    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 19:11:03.667012 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.666988    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 19:11:03.669366 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.669349    2573 manager.go:354] Recovery completed
Sep 10 19:11:03.669588 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.669558    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mvbws"
Sep 10 19:11:03.670658 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.670637    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 10 19:11:03.673683 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.673670    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:11:03.675561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.675542    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:11:03.675630 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.675579    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:11:03.675630 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.675591    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:11:03.675697 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.675543    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mvbws"
Sep 10 19:11:03.676191 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.676176    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 19:11:03.676191 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.676189    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 19:11:03.676191 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.676193    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:11:03.676363 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.676204    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 19:11:03.677013 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.676996    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:11:03.677099 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.677027    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:11:03.677099 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.677041    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:11:03.678708 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.678694    2573 policy_none.go:49] "None policy: Start"
Sep 10 19:11:03.678708 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.678710    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 19:11:03.678802 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.678719    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 19:11:03.680790 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.680776    2573 policy_none.go:47] "Start"
Sep 10 19:11:03.715884 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.715867    2573 manager.go:337] "Starting Device Plugin manager"
Sep 10 19:11:03.737929 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.715910    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 19:11:03.737929 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.715920    2573 server.go:87] "Starting device plugin registration server"
Sep 10 19:11:03.737929 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.716309    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 19:11:03.737929 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.716325    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 19:11:03.737929 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.716407    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 19:11:03.737929 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.716481    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 19:11:03.737929 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.716488    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 19:11:03.737929 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.716989    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 19:11:03.737929 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.717028    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:03.754608 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.754572    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 19:11:03.755936 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.755923    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 19:11:03.756012 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.755965    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 19:11:03.756012 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.755988    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 19:11:03.756109 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.756026    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 19:11:03.758762 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.758743    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:11:03.817122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.817011    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:11:03.818036 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.818017    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:11:03.818125 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.818055    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:11:03.818125 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.818070    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:11:03.818125 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.818095    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-45.ec2.internal"
Sep 10 19:11:03.827135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.827115    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-45.ec2.internal"
Sep 10 19:11:03.827189 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.827141    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-45.ec2.internal\": node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:03.841595 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.841566    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:03.856883 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.856857    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"]
Sep 10 19:11:03.856956 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.856950    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:11:03.857860 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.857843    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:11:03.857938 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.857874    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:11:03.857938 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.857885    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:11:03.860321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.860308    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:11:03.860449 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.860434    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:03.860503 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.860473    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:11:03.861034 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.861020    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:11:03.861078 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.861048    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:11:03.861078 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.861062    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:11:03.861188 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.861061    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:11:03.861223 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.861212    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:11:03.861275 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.861224    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:11:03.863466 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.863451    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:03.863509 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.863485    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:11:03.864158 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.864142    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:11:03.864203 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.864171    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:11:03.864203 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.864180    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:11:03.876819 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.876794    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-45.ec2.internal\" not found" node="ip-10-0-141-45.ec2.internal"
Sep 10 19:11:03.881038 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.881022    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-45.ec2.internal\" not found" node="ip-10-0-141-45.ec2.internal"
Sep 10 19:11:03.941960 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:03.941932    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:03.959500 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.959469    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72f6adf29d1c71bfdc358a4f97e0872d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"72f6adf29d1c71bfdc358a4f97e0872d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:03.959642 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:03.959531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f6adf29d1c71bfdc358a4f97e0872d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"72f6adf29d1c71bfdc358a4f97e0872d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.042824 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.042774    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:04.060082 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.060056    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72f6adf29d1c71bfdc358a4f97e0872d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"72f6adf29d1c71bfdc358a4f97e0872d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.060165 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.060087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f6adf29d1c71bfdc358a4f97e0872d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"72f6adf29d1c71bfdc358a4f97e0872d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.060165 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.060106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cbe649ca88acb6ca8918c435c58b662-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"6cbe649ca88acb6ca8918c435c58b662\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.060290 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.060184    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72f6adf29d1c71bfdc358a4f97e0872d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"72f6adf29d1c71bfdc358a4f97e0872d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.060290 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.060186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72f6adf29d1c71bfdc358a4f97e0872d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal\" (UID: \"72f6adf29d1c71bfdc358a4f97e0872d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.143557 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.143483    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:04.160821 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.160790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cbe649ca88acb6ca8918c435c58b662-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"6cbe649ca88acb6ca8918c435c58b662\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.160900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.160867    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6cbe649ca88acb6ca8918c435c58b662-config\") pod \"kube-apiserver-proxy-ip-10-0-141-45.ec2.internal\" (UID: \"6cbe649ca88acb6ca8918c435c58b662\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.179933 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.179911    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.183685 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.183666    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.244541 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.244504    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:04.345007 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.344976    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:04.445567 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.445495    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-45.ec2.internal\" not found"
Sep 10 19:11:04.506118 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.506091    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:11:04.558123 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.558101    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.571282 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.571246    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 19:11:04.573235 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.573211    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal"
Sep 10 19:11:04.583693 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.583672    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 19:11:04.590853 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.590836    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 19:11:04.590960 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.590944    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 19:11:04.591024 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.590980    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 19:11:04.633180 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.633149    2573 apiserver.go:52] "Watching apiserver"
Sep 10 19:11:04.654596 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.654571    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 19:11:04.654981 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.654959    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-5rfzq","openshift-multus/network-metrics-daemon-5v6bv","openshift-network-operator/iptables-alerter-7r4lv","kube-system/konnectivity-agent-wn7z8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv","openshift-dns/node-resolver-mbsgt","openshift-multus/multus-additional-cni-plugins-967gq","openshift-network-diagnostics/network-check-target-qlmln","openshift-ovn-kubernetes/ovnkube-node-wxrqg","kube-system/global-pull-secret-syncer-htz6q","kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal","openshift-cluster-node-tuning-operator/tuned-cxb2v","openshift-image-registry/node-ca-ctj27","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal"]
Sep 10 19:11:04.657466 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.657448    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 19:11:04.657598 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.657578    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.659716 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.659697    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:04.659804 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.659752    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:04.659986 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.659972    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 19:11:04.660271 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.660237    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz88g\""
Sep 10 19:11:04.660327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.660313    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 19:11:04.660847 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.660832    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 19:11:04.661935 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.661917    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.662946 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.662923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9tv6\" (UniqueName: \"kubernetes.io/projected/22c1eacb-21a4-45db-ba0a-5e727cb382e3-kube-api-access-j9tv6\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.663032 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.662967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:04.663032 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.662999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-socket-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.663032 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.663026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-device-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.663182 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.663050    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.663182 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.663079    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lghhl\" (UniqueName: \"kubernetes.io/projected/46be1c16-0b44-44ae-8675-69b7b8442231-kube-api-access-lghhl\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:04.663182 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.663109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.663182 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.663144    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-registration-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.663182 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.663172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-sys-fs\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.664762 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.664741    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:04.664977 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.664749    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 19:11:04.664977 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.664947    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 19:11:04.665178 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.665167    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ckj79\""
Sep 10 19:11:04.665274 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.665217    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 19:11:04.665274 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.665266    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 19:11:04.665466 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.665454    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 19:11:04.666895 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.666879    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.667096 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.667078    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 19:11:04.667199 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.667087    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f85vw\""
Sep 10 19:11:04.667199 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.667101    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 19:11:04.668507 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.668488    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 19:11:04.669135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.669109    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:11:04.669230 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.669189    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.669230 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.669218    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 19:11:04.669373 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.669272    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 19:11:04.669428 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.669390    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gwx4d\""
Sep 10 19:11:04.671363 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.671340    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n2w68\""
Sep 10 19:11:04.671449 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.671429    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.671552 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.671535    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 19:11:04.671646 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.671574    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 19:11:04.673619 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.673603    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:04.673711 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.673694    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 19:11:04.673766 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.673711    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:04.673955 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.673943    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-44sd9\""
Sep 10 19:11:04.675964 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.675949    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.678205 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.678164    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 19:06:04 +0000 UTC" deadline="2028-04-16 14:07:30.397087059 +0000 UTC"
Sep 10 19:11:04.678321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.678211    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14010h56m25.718880033s"
Sep 10 19:11:04.678321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.678195    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:04.678403 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.678319    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:04.679448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.679435    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 19:11:04.679520 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.679460    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 19:11:04.679520 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.679484    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5zgxn\""
Sep 10 19:11:04.679621 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.679547    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 19:11:04.679621 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.679557    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 19:11:04.679621 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.679568    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 19:11:04.679719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.679680    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 19:11:04.680569 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.680553    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.682752 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.682719    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 19:11:04.682837 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.682750    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:11:04.682837 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.682751    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.682991 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.682975    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kg4jk\""
Sep 10 19:11:04.685086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.685065    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 19:11:04.685179 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.685088    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 19:11:04.685179 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.685096    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c7k4l\""
Sep 10 19:11:04.685179 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.685094    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 19:11:04.700692 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.700639    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5n9sq"
Sep 10 19:11:04.708584 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.708567    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5n9sq"
Sep 10 19:11:04.759767 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.759741    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 19:11:04.763343 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lghhl\" (UniqueName: \"kubernetes.io/projected/46be1c16-0b44-44ae-8675-69b7b8442231-kube-api-access-lghhl\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:04.763513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763349    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f1754fb-fef4-4250-bffa-2032c9acdd11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.763513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763365    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltdq\" (UniqueName: \"kubernetes.io/projected/0f1754fb-fef4-4250-bffa-2032c9acdd11-kube-api-access-nltdq\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.763513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763384    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-etc-openvswitch\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.763513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-cni-netd\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.763513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763416    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-systemd\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.763513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f1754fb-fef4-4250-bffa-2032c9acdd11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.763513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjtjz\" (UniqueName: \"kubernetes.io/projected/f1c4d93c-dd02-4ea0-b085-97afdf95ac47-kube-api-access-xjtjz\") pod \"iptables-alerter-7r4lv\" (UID: \"f1c4d93c-dd02-4ea0-b085-97afdf95ac47\") " pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763513    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-daemon-config\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763548    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-etc-kubernetes\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763572    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-kubelet\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763593    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-log-socket\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b8ab082-b524-4dc7-b4a2-73f54fb82423-ovn-node-metrics-cert\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763638    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-node-log\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763686    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f1754fb-fef4-4250-bffa-2032c9acdd11-system-cni-dir\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763699    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-system-cni-dir\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-run-k8s-cni-cncf-io\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-slash\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763736    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763752    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xxp\" (UniqueName: \"kubernetes.io/projected/0b8ab082-b524-4dc7-b4a2-73f54fb82423-kube-api-access-d8xxp\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.763784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763774    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-modprobe-d\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763792    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1c4d93c-dd02-4ea0-b085-97afdf95ac47-iptables-alerter-script\") pod \"iptables-alerter-7r4lv\" (UID: \"f1c4d93c-dd02-4ea0-b085-97afdf95ac47\") " pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-run-netns\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-run-multus-certs\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-var-lib-kubelet\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-tuned\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763891    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-sys-fs\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763906    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1c4d93c-dd02-4ea0-b085-97afdf95ac47-host-slash\") pod \"iptables-alerter-7r4lv\" (UID: \"f1c4d93c-dd02-4ea0-b085-97afdf95ac47\") " pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763919    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-kubernetes\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-var-lib-kubelet\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763941    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-sys-fs\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.763959    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-systemd-units\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-run-ovn\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-cni-bin\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764061    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764083    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b8ab082-b524-4dc7-b4a2-73f54fb82423-ovnkube-script-lib\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.764216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-registration-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764128    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9tv6\" (UniqueName: \"kubernetes.io/projected/22c1eacb-21a4-45db-ba0a-5e727cb382e3-kube-api-access-j9tv6\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764142    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-cnibin\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764155    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-run-systemd\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764176    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b8ab082-b524-4dc7-b4a2-73f54fb82423-ovnkube-config\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-registration-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764192    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-kubelet-config\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764215    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764234    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-host\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-var-lib-cni-bin\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-cni-dir\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764339    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-run\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b-agent-certs\") pod \"konnectivity-agent-wn7z8\" (UID: \"356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b\") " pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764373    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-socket-dir-parent\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b8ab082-b524-4dc7-b4a2-73f54fb82423-env-overrides\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-sysctl-d\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.764686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0aa983d7-eafe-4eb0-9cf1-b236e928a649-cni-binary-copy\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-conf-dir\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764488    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxw2v\" (UniqueName: \"kubernetes.io/projected/0aa983d7-eafe-4eb0-9cf1-b236e928a649-kube-api-access-qxw2v\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.764489    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.764560    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs podName:46be1c16-0b44-44ae-8675-69b7b8442231 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:05.264536687 +0000 UTC m=+1.995232575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs") pod "network-metrics-daemon-5v6bv" (UID: "46be1c16-0b44-44ae-8675-69b7b8442231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d5123b1-49ac-4bcc-99d3-ef530a387f86-host\") pod \"node-ca-ctj27\" (UID: \"4d5123b1-49ac-4bcc-99d3-ef530a387f86\") " pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764599    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq74j\" (UniqueName: \"kubernetes.io/projected/4d5123b1-49ac-4bcc-99d3-ef530a387f86-kube-api-access-dq74j\") pod \"node-ca-ctj27\" (UID: \"4d5123b1-49ac-4bcc-99d3-ef530a387f86\") " pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764613    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbf8a58b-9706-4782-acc4-69500bedc6fc-tmp\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-socket-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-device-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764672    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764687    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f1754fb-fef4-4250-bffa-2032c9acdd11-cnibin\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f1754fb-fef4-4250-bffa-2032c9acdd11-cni-binary-copy\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764750    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-run-netns\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.765342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764762    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764776    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-socket-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-sys\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8c22ae-6623-4996-a91f-1007228beb5e-tmp-dir\") pod \"node-resolver-mbsgt\" (UID: \"cc8c22ae-6623-4996-a91f-1007228beb5e\") " pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62dm\" (UniqueName: \"kubernetes.io/projected/cc8c22ae-6623-4996-a91f-1007228beb5e-kube-api-access-s62dm\") pod \"node-resolver-mbsgt\" (UID: \"cc8c22ae-6623-4996-a91f-1007228beb5e\") " pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-lib-modules\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764890    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4wx8\" (UniqueName: \"kubernetes.io/projected/dbf8a58b-9706-4782-acc4-69500bedc6fc-kube-api-access-v4wx8\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764895    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/22c1eacb-21a4-45db-ba0a-5e727cb382e3-device-dir\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764906    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc8c22ae-6623-4996-a91f-1007228beb5e-hosts-file\") pod \"node-resolver-mbsgt\" (UID: \"cc8c22ae-6623-4996-a91f-1007228beb5e\") " pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764930    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-var-lib-cni-multus\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764954    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-var-lib-openvswitch\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-dbus\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.764995    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-sysconfig\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.765009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-sysctl-conf\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.765023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f1754fb-fef4-4250-bffa-2032c9acdd11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.765038    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b-konnectivity-ca\") pod \"konnectivity-agent-wn7z8\" (UID: \"356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b\") " pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.765060    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-hostroot\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.765803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.765087    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d5123b1-49ac-4bcc-99d3-ef530a387f86-serviceca\") pod \"node-ca-ctj27\" (UID: \"4d5123b1-49ac-4bcc-99d3-ef530a387f86\") " pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.766293 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.765103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-run-openvswitch\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.770862 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.770844    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 19:11:04.774410 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.774386    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9tv6\" (UniqueName: \"kubernetes.io/projected/22c1eacb-21a4-45db-ba0a-5e727cb382e3-kube-api-access-j9tv6\") pod \"aws-ebs-csi-driver-node-x87tv\" (UID: \"22c1eacb-21a4-45db-ba0a-5e727cb382e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.774496 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.774463    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lghhl\" (UniqueName: \"kubernetes.io/projected/46be1c16-0b44-44ae-8675-69b7b8442231-kube-api-access-lghhl\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:04.865591 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltdq\" (UniqueName: \"kubernetes.io/projected/0f1754fb-fef4-4250-bffa-2032c9acdd11-kube-api-access-nltdq\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.865591 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-etc-openvswitch\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-cni-netd\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-systemd\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f1754fb-fef4-4250-bffa-2032c9acdd11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865649    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtjz\" (UniqueName: \"kubernetes.io/projected/f1c4d93c-dd02-4ea0-b085-97afdf95ac47-kube-api-access-xjtjz\") pod \"iptables-alerter-7r4lv\" (UID: \"f1c4d93c-dd02-4ea0-b085-97afdf95ac47\") " pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-daemon-config\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865693    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-etc-openvswitch\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865728    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-systemd\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865704    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-etc-kubernetes\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865751    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-etc-kubernetes\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865766    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-kubelet\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865693    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-cni-netd\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865789    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-log-socket\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865795    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-kubelet\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865813    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b8ab082-b524-4dc7-b4a2-73f54fb82423-ovn-node-metrics-cert\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-node-log\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865833    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-log-socket\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.865839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f1754fb-fef4-4250-bffa-2032c9acdd11-system-cni-dir\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-system-cni-dir\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865889    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-run-k8s-cni-cncf-io\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865893    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-node-log\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-slash\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xxp\" (UniqueName: \"kubernetes.io/projected/0b8ab082-b524-4dc7-b4a2-73f54fb82423-kube-api-access-d8xxp\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865934    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f1754fb-fef4-4250-bffa-2032c9acdd11-system-cni-dir\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-modprobe-d\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.865999    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-run-k8s-cni-cncf-io\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866011    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-system-cni-dir\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866016    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-slash\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1c4d93c-dd02-4ea0-b085-97afdf95ac47-iptables-alerter-script\") pod \"iptables-alerter-7r4lv\" (UID: \"f1c4d93c-dd02-4ea0-b085-97afdf95ac47\") " pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866124    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-modprobe-d\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866137    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-run-netns\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-run-multus-certs\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866172    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-run-netns\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866208    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-run-multus-certs\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-var-lib-kubelet\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.866665 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866275    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-daemon-config\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866275    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0f1754fb-fef4-4250-bffa-2032c9acdd11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866334    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-var-lib-kubelet\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866282    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-tuned\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866388    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1c4d93c-dd02-4ea0-b085-97afdf95ac47-host-slash\") pod \"iptables-alerter-7r4lv\" (UID: \"f1c4d93c-dd02-4ea0-b085-97afdf95ac47\") " pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866416    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-kubernetes\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-var-lib-kubelet\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866470    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-systemd-units\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866446    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1c4d93c-dd02-4ea0-b085-97afdf95ac47-host-slash\") pod \"iptables-alerter-7r4lv\" (UID: \"f1c4d93c-dd02-4ea0-b085-97afdf95ac47\") " pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866495    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-var-lib-kubelet\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866496    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-run-ovn\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-cni-bin\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866536    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-run-ovn\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866543    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-systemd-units\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866473    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-kubernetes\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866581    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-cni-bin\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.867629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866583    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b8ab082-b524-4dc7-b4a2-73f54fb82423-ovnkube-script-lib\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866595    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866610    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-cnibin\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866631    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-run-systemd\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866649    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b8ab082-b524-4dc7-b4a2-73f54fb82423-ovnkube-config\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-kubelet-config\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-host\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866709    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-var-lib-cni-bin\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866728    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-cni-dir\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-run\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b-agent-certs\") pod \"konnectivity-agent-wn7z8\" (UID: \"356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b\") " pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866836    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-socket-dir-parent\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b8ab082-b524-4dc7-b4a2-73f54fb82423-env-overrides\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-sysctl-d\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866889    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0aa983d7-eafe-4eb0-9cf1-b236e928a649-cni-binary-copy\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866888    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-cnibin\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.868479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-conf-dir\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866928    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxw2v\" (UniqueName: \"kubernetes.io/projected/0aa983d7-eafe-4eb0-9cf1-b236e928a649-kube-api-access-qxw2v\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866961    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-cni-dir\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866970    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d5123b1-49ac-4bcc-99d3-ef530a387f86-host\") pod \"node-ca-ctj27\" (UID: \"4d5123b1-49ac-4bcc-99d3-ef530a387f86\") " pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866979    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-socket-dir-parent\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866995    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-run-systemd\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-kubelet-config\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867093    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-var-lib-cni-bin\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867059    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-run\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867187    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f1c4d93c-dd02-4ea0-b085-97afdf95ac47-iptables-alerter-script\") pod \"iptables-alerter-7r4lv\" (UID: \"f1c4d93c-dd02-4ea0-b085-97afdf95ac47\") " pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.867200    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867162    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d5123b1-49ac-4bcc-99d3-ef530a387f86-host\") pod \"node-ca-ctj27\" (UID: \"4d5123b1-49ac-4bcc-99d3-ef530a387f86\") " pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.867286    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret podName:bfc622a4-f58d-42ec-97f6-5271a1a9d3bc nodeName:}" failed. No retries permitted until 2026-09-10 19:11:05.367267876 +0000 UTC m=+2.097963777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret") pod "global-pull-secret-syncer-htz6q" (UID: "bfc622a4-f58d-42ec-97f6-5271a1a9d3bc") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867303    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-host\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867310    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-sysctl-d\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867059    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-multus-conf-dir\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.866995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq74j\" (UniqueName: \"kubernetes.io/projected/4d5123b1-49ac-4bcc-99d3-ef530a387f86-kube-api-access-dq74j\") pod \"node-ca-ctj27\" (UID: \"4d5123b1-49ac-4bcc-99d3-ef530a387f86\") " pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbf8a58b-9706-4782-acc4-69500bedc6fc-tmp\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.869463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f1754fb-fef4-4250-bffa-2032c9acdd11-cnibin\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867417    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f1754fb-fef4-4250-bffa-2032c9acdd11-cni-binary-copy\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867440    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867462    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-run-netns\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-sys\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867496    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f1754fb-fef4-4250-bffa-2032c9acdd11-cnibin\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867504    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8c22ae-6623-4996-a91f-1007228beb5e-tmp-dir\") pod \"node-resolver-mbsgt\" (UID: \"cc8c22ae-6623-4996-a91f-1007228beb5e\") " pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867547    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62dm\" (UniqueName: \"kubernetes.io/projected/cc8c22ae-6623-4996-a91f-1007228beb5e-kube-api-access-s62dm\") pod \"node-resolver-mbsgt\" (UID: \"cc8c22ae-6623-4996-a91f-1007228beb5e\") " pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867552    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0aa983d7-eafe-4eb0-9cf1-b236e928a649-cni-binary-copy\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-lib-modules\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867601    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867603    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4wx8\" (UniqueName: \"kubernetes.io/projected/dbf8a58b-9706-4782-acc4-69500bedc6fc-kube-api-access-v4wx8\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867646    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc8c22ae-6623-4996-a91f-1007228beb5e-hosts-file\") pod \"node-resolver-mbsgt\" (UID: \"cc8c22ae-6623-4996-a91f-1007228beb5e\") " pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867671    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-var-lib-cni-multus\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-var-lib-openvswitch\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-dbus\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867742    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-sysconfig\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.870694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc8c22ae-6623-4996-a91f-1007228beb5e-tmp-dir\") pod \"node-resolver-mbsgt\" (UID: \"cc8c22ae-6623-4996-a91f-1007228beb5e\") " pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867765    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-sysctl-conf\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f1754fb-fef4-4250-bffa-2032c9acdd11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b-konnectivity-ca\") pod \"konnectivity-agent-wn7z8\" (UID: \"356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b\") " pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867843    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-hostroot\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867861    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-host-run-netns\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867865    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d5123b1-49ac-4bcc-99d3-ef530a387f86-serviceca\") pod \"node-ca-ctj27\" (UID: \"4d5123b1-49ac-4bcc-99d3-ef530a387f86\") " pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.867895    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-run-openvswitch\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868059    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-hostroot\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868149    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-sys\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f1754fb-fef4-4250-bffa-2032c9acdd11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868409    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f1754fb-fef4-4250-bffa-2032c9acdd11-cni-binary-copy\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868437    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f1754fb-fef4-4250-bffa-2032c9acdd11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868474    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-sysconfig\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-run-openvswitch\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868549    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cc8c22ae-6623-4996-a91f-1007228beb5e-hosts-file\") pod \"node-resolver-mbsgt\" (UID: \"cc8c22ae-6623-4996-a91f-1007228beb5e\") " pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868580    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0aa983d7-eafe-4eb0-9cf1-b236e928a649-host-var-lib-cni-multus\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868611    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b8ab082-b524-4dc7-b4a2-73f54fb82423-var-lib-openvswitch\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.871457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868653    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-dbus\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.868964    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-sysctl-conf\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.869043    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbf8a58b-9706-4782-acc4-69500bedc6fc-lib-modules\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.869082    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0f1754fb-fef4-4250-bffa-2032c9acdd11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.869170    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b8ab082-b524-4dc7-b4a2-73f54fb82423-ovnkube-script-lib\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.869350    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4d5123b1-49ac-4bcc-99d3-ef530a387f86-serviceca\") pod \"node-ca-ctj27\" (UID: \"4d5123b1-49ac-4bcc-99d3-ef530a387f86\") " pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.869756    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbf8a58b-9706-4782-acc4-69500bedc6fc-etc-tuned\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.870160    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b8ab082-b524-4dc7-b4a2-73f54fb82423-ovn-node-metrics-cert\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.870290    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b8ab082-b524-4dc7-b4a2-73f54fb82423-env-overrides\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.870377    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b8ab082-b524-4dc7-b4a2-73f54fb82423-ovnkube-config\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.870878    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b-konnectivity-ca\") pod \"konnectivity-agent-wn7z8\" (UID: \"356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b\") " pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.870911    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbf8a58b-9706-4782-acc4-69500bedc6fc-tmp\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.871945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.871125    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b-agent-certs\") pod \"konnectivity-agent-wn7z8\" (UID: \"356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b\") " pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:04.875170 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.875148    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:11:04.875357 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.875177    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:11:04.875357 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.875191    2573 projected.go:196] Error preparing data for projected volume kube-api-access-67hfl for pod openshift-network-diagnostics/network-check-target-qlmln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:04.875357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.875200    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xxp\" (UniqueName: \"kubernetes.io/projected/0b8ab082-b524-4dc7-b4a2-73f54fb82423-kube-api-access-d8xxp\") pod \"ovnkube-node-wxrqg\" (UID: \"0b8ab082-b524-4dc7-b4a2-73f54fb82423\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:04.875357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.875204    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltdq\" (UniqueName: \"kubernetes.io/projected/0f1754fb-fef4-4250-bffa-2032c9acdd11-kube-api-access-nltdq\") pod \"multus-additional-cni-plugins-967gq\" (UID: \"0f1754fb-fef4-4250-bffa-2032c9acdd11\") " pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:04.875357 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:04.875281    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl podName:c15a2707-8b56-4c41-8d37-98cd53e9d4e6 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:05.375267895 +0000 UTC m=+2.105963801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-67hfl" (UniqueName: "kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl") pod "network-check-target-qlmln" (UID: "c15a2707-8b56-4c41-8d37-98cd53e9d4e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:04.876053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.876031    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtjz\" (UniqueName: \"kubernetes.io/projected/f1c4d93c-dd02-4ea0-b085-97afdf95ac47-kube-api-access-xjtjz\") pod \"iptables-alerter-7r4lv\" (UID: \"f1c4d93c-dd02-4ea0-b085-97afdf95ac47\") " pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:04.877048 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.877030    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq74j\" (UniqueName: \"kubernetes.io/projected/4d5123b1-49ac-4bcc-99d3-ef530a387f86-kube-api-access-dq74j\") pod \"node-ca-ctj27\" (UID: \"4d5123b1-49ac-4bcc-99d3-ef530a387f86\") " pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:04.880343 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.880320    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62dm\" (UniqueName: \"kubernetes.io/projected/cc8c22ae-6623-4996-a91f-1007228beb5e-kube-api-access-s62dm\") pod \"node-resolver-mbsgt\" (UID: \"cc8c22ae-6623-4996-a91f-1007228beb5e\") " pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:04.881175 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.881159    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxw2v\" (UniqueName: \"kubernetes.io/projected/0aa983d7-eafe-4eb0-9cf1-b236e928a649-kube-api-access-qxw2v\") pod \"multus-5rfzq\" (UID: \"0aa983d7-eafe-4eb0-9cf1-b236e928a649\") " pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:04.881756 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.881737    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4wx8\" (UniqueName: \"kubernetes.io/projected/dbf8a58b-9706-4782-acc4-69500bedc6fc-kube-api-access-v4wx8\") pod \"tuned-cxb2v\" (UID: \"dbf8a58b-9706-4782-acc4-69500bedc6fc\") " pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:04.899092 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:04.899065    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f6adf29d1c71bfdc358a4f97e0872d.slice/crio-7f18b1262bb9881f57a0a9cfda4692c70987c2953f1ab77d405184f9e2c3daa0 WatchSource:0}: Error finding container 7f18b1262bb9881f57a0a9cfda4692c70987c2953f1ab77d405184f9e2c3daa0: Status 404 returned error can't find the container with id 7f18b1262bb9881f57a0a9cfda4692c70987c2953f1ab77d405184f9e2c3daa0
Sep 10 19:11:04.899541 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:04.899520    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbe649ca88acb6ca8918c435c58b662.slice/crio-338aaa5110b9105755d605ede63461ebc24e27380b273d7a2945cad7b915e11c WatchSource:0}: Error finding container 338aaa5110b9105755d605ede63461ebc24e27380b273d7a2945cad7b915e11c: Status 404 returned error can't find the container with id 338aaa5110b9105755d605ede63461ebc24e27380b273d7a2945cad7b915e11c
Sep 10 19:11:04.903537 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.903521    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:11:04.945768 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.945741    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:11:04.987420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.987338    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv"
Sep 10 19:11:04.997649 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:04.997626    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c1eacb_21a4_45db_ba0a_5e727cb382e3.slice/crio-d4c98f6f8bb9ef288db1d48341e66378dda5f090730669e6d60a4241d62bcf2f WatchSource:0}: Error finding container d4c98f6f8bb9ef288db1d48341e66378dda5f090730669e6d60a4241d62bcf2f: Status 404 returned error can't find the container with id d4c98f6f8bb9ef288db1d48341e66378dda5f090730669e6d60a4241d62bcf2f
Sep 10 19:11:04.998243 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:04.998222    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-967gq"
Sep 10 19:11:06.077809 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.077788    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:06.078104 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:06.078082    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1754fb_fef4_4250_bffa_2032c9acdd11.slice/crio-ff65f5cbde7674e5189c30ffe7cefbed0e5aa0e34427eb727208562ba688b743 WatchSource:0}: Error finding container ff65f5cbde7674e5189c30ffe7cefbed0e5aa0e34427eb727208562ba688b743: Status 404 returned error can't find the container with id ff65f5cbde7674e5189c30ffe7cefbed0e5aa0e34427eb727208562ba688b743
Sep 10 19:11:06.086838 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:06.086817    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356a1f2e_3fbb_48a4_b9a1_b8739eef5b8b.slice/crio-dc696df76833c6dffcad3162e6f32d90855785fb7aec5212a148d221605be620 WatchSource:0}: Error finding container dc696df76833c6dffcad3162e6f32d90855785fb7aec5212a148d221605be620: Status 404 returned error can't find the container with id dc696df76833c6dffcad3162e6f32d90855785fb7aec5212a148d221605be620
Sep 10 19:11:06.112625 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.112584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7r4lv"
Sep 10 19:11:06.120777 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:06.120753    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c4d93c_dd02_4ea0_b085_97afdf95ac47.slice/crio-2498dbafc0802880e072c4b106f7abfdcb0b6a75fe76af23aaa79e894c32b550 WatchSource:0}: Error finding container 2498dbafc0802880e072c4b106f7abfdcb0b6a75fe76af23aaa79e894c32b550: Status 404 returned error can't find the container with id 2498dbafc0802880e072c4b106f7abfdcb0b6a75fe76af23aaa79e894c32b550
Sep 10 19:11:06.128707 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.128687    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mbsgt"
Sep 10 19:11:06.133449 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.133433    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5rfzq"
Sep 10 19:11:06.136707 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:06.136688    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8c22ae_6623_4996_a91f_1007228beb5e.slice/crio-433024ba585aa01e43f4a16f44ea5a3c4456215a5660feadcc8dde13e93e613a WatchSource:0}: Error finding container 433024ba585aa01e43f4a16f44ea5a3c4456215a5660feadcc8dde13e93e613a: Status 404 returned error can't find the container with id 433024ba585aa01e43f4a16f44ea5a3c4456215a5660feadcc8dde13e93e613a
Sep 10 19:11:06.141742 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:06.141725    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa983d7_eafe_4eb0_9cf1_b236e928a649.slice/crio-220b8522c9ebbf9b36fa9067344d11d7b35938b81fbe548f354917be5813ed63 WatchSource:0}: Error finding container 220b8522c9ebbf9b36fa9067344d11d7b35938b81fbe548f354917be5813ed63: Status 404 returned error can't find the container with id 220b8522c9ebbf9b36fa9067344d11d7b35938b81fbe548f354917be5813ed63
Sep 10 19:11:06.157642 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.157624    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:06.166579 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:06.166559    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8ab082_b524_4dc7_b4a2_73f54fb82423.slice/crio-30ad41fb65f7edb13bc268a6a523c6dc0373c27a1482844f3c1c2fc18ea70dcb WatchSource:0}: Error finding container 30ad41fb65f7edb13bc268a6a523c6dc0373c27a1482844f3c1c2fc18ea70dcb: Status 404 returned error can't find the container with id 30ad41fb65f7edb13bc268a6a523c6dc0373c27a1482844f3c1c2fc18ea70dcb
Sep 10 19:11:06.171651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.171636    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cxb2v"
Sep 10 19:11:06.176497 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.176477    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctj27"
Sep 10 19:11:06.179332 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:06.179313    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf8a58b_9706_4782_acc4_69500bedc6fc.slice/crio-9b71cc10a2e051ddd7e5988bb229e1a1d3da270161fe9eff5d1e80ce6ba73d09 WatchSource:0}: Error finding container 9b71cc10a2e051ddd7e5988bb229e1a1d3da270161fe9eff5d1e80ce6ba73d09: Status 404 returned error can't find the container with id 9b71cc10a2e051ddd7e5988bb229e1a1d3da270161fe9eff5d1e80ce6ba73d09
Sep 10 19:11:06.182900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.182660    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:11:06.185538 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:06.185518    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5123b1_49ac_4bcc_99d3_ef530a387f86.slice/crio-64cc8df7de4589072aa32bc741d32e70a89730b46de029a68748ba2683c7fcf6 WatchSource:0}: Error finding container 64cc8df7de4589072aa32bc741d32e70a89730b46de029a68748ba2683c7fcf6: Status 404 returned error can't find the container with id 64cc8df7de4589072aa32bc741d32e70a89730b46de029a68748ba2683c7fcf6
Sep 10 19:11:06.342353 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.342272    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:06.342485 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:06.342384    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:06.342485 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:06.342433    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs podName:46be1c16-0b44-44ae-8675-69b7b8442231 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:07.342418762 +0000 UTC m=+3.002215945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs") pod "network-metrics-daemon-5v6bv" (UID: "46be1c16-0b44-44ae-8675-69b7b8442231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:06.443380 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.443341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:06.443522 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:06.443450    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:06.443522 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:06.443497    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret podName:bfc622a4-f58d-42ec-97f6-5271a1a9d3bc nodeName:}" failed. No retries permitted until 2026-09-10 19:11:07.443484723 +0000 UTC m=+3.103281906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret") pod "global-pull-secret-syncer-htz6q" (UID: "bfc622a4-f58d-42ec-97f6-5271a1a9d3bc") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:06.544401 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.544371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:06.544538 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:06.544509    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:11:06.544538 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:06.544531    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:11:06.544607 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:06.544540    2573 projected.go:196] Error preparing data for projected volume kube-api-access-67hfl for pod openshift-network-diagnostics/network-check-target-qlmln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:06.544607 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:06.544596    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl podName:c15a2707-8b56-4c41-8d37-98cd53e9d4e6 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:07.544579312 +0000 UTC m=+3.204376483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-67hfl" (UniqueName: "kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl") pod "network-check-target-qlmln" (UID: "c15a2707-8b56-4c41-8d37-98cd53e9d4e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:06.782834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.782719    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:06:05 +0000 UTC" deadline="2028-04-14 17:58:09.615718833 +0000 UTC"
Sep 10 19:11:06.782834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.782783    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13966h47m2.832940235s"
Sep 10 19:11:06.850387 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.850138    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctj27" event={"ID":"4d5123b1-49ac-4bcc-99d3-ef530a387f86","Type":"ContainerStarted","Data":"64cc8df7de4589072aa32bc741d32e70a89730b46de029a68748ba2683c7fcf6"}
Sep 10 19:11:06.861907 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.861869    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cxb2v" event={"ID":"dbf8a58b-9706-4782-acc4-69500bedc6fc","Type":"ContainerStarted","Data":"9b71cc10a2e051ddd7e5988bb229e1a1d3da270161fe9eff5d1e80ce6ba73d09"}
Sep 10 19:11:06.875926 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.875830    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" event={"ID":"0b8ab082-b524-4dc7-b4a2-73f54fb82423","Type":"ContainerStarted","Data":"30ad41fb65f7edb13bc268a6a523c6dc0373c27a1482844f3c1c2fc18ea70dcb"}
Sep 10 19:11:06.881523 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.881491    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbsgt" event={"ID":"cc8c22ae-6623-4996-a91f-1007228beb5e","Type":"ContainerStarted","Data":"433024ba585aa01e43f4a16f44ea5a3c4456215a5660feadcc8dde13e93e613a"}
Sep 10 19:11:06.884806 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.884780    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-967gq" event={"ID":"0f1754fb-fef4-4250-bffa-2032c9acdd11","Type":"ContainerStarted","Data":"ff65f5cbde7674e5189c30ffe7cefbed0e5aa0e34427eb727208562ba688b743"}
Sep 10 19:11:06.893249 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.893220    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" event={"ID":"6cbe649ca88acb6ca8918c435c58b662","Type":"ContainerStarted","Data":"338aaa5110b9105755d605ede63461ebc24e27380b273d7a2945cad7b915e11c"}
Sep 10 19:11:06.896694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.896668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"72f6adf29d1c71bfdc358a4f97e0872d","Type":"ContainerStarted","Data":"7f18b1262bb9881f57a0a9cfda4692c70987c2953f1ab77d405184f9e2c3daa0"}
Sep 10 19:11:06.907369 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.907336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5rfzq" event={"ID":"0aa983d7-eafe-4eb0-9cf1-b236e928a649","Type":"ContainerStarted","Data":"220b8522c9ebbf9b36fa9067344d11d7b35938b81fbe548f354917be5813ed63"}
Sep 10 19:11:06.914413 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.914379    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7r4lv" event={"ID":"f1c4d93c-dd02-4ea0-b085-97afdf95ac47","Type":"ContainerStarted","Data":"2498dbafc0802880e072c4b106f7abfdcb0b6a75fe76af23aaa79e894c32b550"}
Sep 10 19:11:06.921421 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.921368    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wn7z8" event={"ID":"356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b","Type":"ContainerStarted","Data":"dc696df76833c6dffcad3162e6f32d90855785fb7aec5212a148d221605be620"}
Sep 10 19:11:06.926355 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:06.926327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv" event={"ID":"22c1eacb-21a4-45db-ba0a-5e727cb382e3","Type":"ContainerStarted","Data":"d4c98f6f8bb9ef288db1d48341e66378dda5f090730669e6d60a4241d62bcf2f"}
Sep 10 19:11:07.178395 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:07.178321    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:11:07.351610 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:07.351576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:07.351800 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.351735    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:07.351870 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.351803    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs podName:46be1c16-0b44-44ae-8675-69b7b8442231 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:09.351783713 +0000 UTC m=+5.011580886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs") pod "network-metrics-daemon-5v6bv" (UID: "46be1c16-0b44-44ae-8675-69b7b8442231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:07.453032 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:07.452945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:07.453219 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.453078    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:07.453219 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.453143    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret podName:bfc622a4-f58d-42ec-97f6-5271a1a9d3bc nodeName:}" failed. No retries permitted until 2026-09-10 19:11:09.453125137 +0000 UTC m=+5.112922311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret") pod "global-pull-secret-syncer-htz6q" (UID: "bfc622a4-f58d-42ec-97f6-5271a1a9d3bc") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:07.553787 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:07.553743    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:07.553977 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.553955    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:11:07.553977 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.553977    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:11:07.554105 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.553990    2573 projected.go:196] Error preparing data for projected volume kube-api-access-67hfl for pod openshift-network-diagnostics/network-check-target-qlmln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:07.554105 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.554052    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl podName:c15a2707-8b56-4c41-8d37-98cd53e9d4e6 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:09.554033729 +0000 UTC m=+5.213830919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-67hfl" (UniqueName: "kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl") pod "network-check-target-qlmln" (UID: "c15a2707-8b56-4c41-8d37-98cd53e9d4e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:07.783290 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:07.783187    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:06:05 +0000 UTC" deadline="2028-05-04 17:21:15.035526938 +0000 UTC"
Sep 10 19:11:07.783290 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:07.783233    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14446h10m7.252299124s"
Sep 10 19:11:07.827874 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:07.827813    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:07.828038 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.827949    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:07.828119 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:07.828034    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:07.828119 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.828110    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:07.828257 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:07.828193    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:07.828308 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:07.828277    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:08.262739 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:08.262652    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:11:09.372612 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:09.372549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:09.373179 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.372831    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:09.373179 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.372929    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs podName:46be1c16-0b44-44ae-8675-69b7b8442231 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:13.372883474 +0000 UTC m=+9.032680661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs") pod "network-metrics-daemon-5v6bv" (UID: "46be1c16-0b44-44ae-8675-69b7b8442231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:09.474063 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:09.474019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:09.474271 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.474190    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:09.474271 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.474264    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret podName:bfc622a4-f58d-42ec-97f6-5271a1a9d3bc nodeName:}" failed. No retries permitted until 2026-09-10 19:11:13.474243449 +0000 UTC m=+9.134040637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret") pod "global-pull-secret-syncer-htz6q" (UID: "bfc622a4-f58d-42ec-97f6-5271a1a9d3bc") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:09.574766 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:09.574731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:09.574937 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.574886    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:11:09.574937 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.574906    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:11:09.574937 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.574918    2573 projected.go:196] Error preparing data for projected volume kube-api-access-67hfl for pod openshift-network-diagnostics/network-check-target-qlmln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:09.575090 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.574974    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl podName:c15a2707-8b56-4c41-8d37-98cd53e9d4e6 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:13.574957427 +0000 UTC m=+9.234754599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-67hfl" (UniqueName: "kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl") pod "network-check-target-qlmln" (UID: "c15a2707-8b56-4c41-8d37-98cd53e9d4e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:09.827525 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:09.827441    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:09.827697 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:09.827572    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:09.827697 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.827586    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:09.827821 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.827702    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:09.827821 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:09.827785    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:09.827904 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:09.827869    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:11.828178 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:11.828077    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:11.828178 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:11.828077    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:11.828750 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:11.828230    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:11.828750 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:11.828318    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:11.828750 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:11.828389    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:11.828750 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:11.828468    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:13.409183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:13.409124    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:13.409711 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.409309    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:13.409711 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.409394    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs podName:46be1c16-0b44-44ae-8675-69b7b8442231 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:21.409372372 +0000 UTC m=+17.069169548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs") pod "network-metrics-daemon-5v6bv" (UID: "46be1c16-0b44-44ae-8675-69b7b8442231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:13.509680 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:13.509641    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:13.509869 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.509846    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:13.509943 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.509929    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret podName:bfc622a4-f58d-42ec-97f6-5271a1a9d3bc nodeName:}" failed. No retries permitted until 2026-09-10 19:11:21.509908248 +0000 UTC m=+17.169705439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret") pod "global-pull-secret-syncer-htz6q" (UID: "bfc622a4-f58d-42ec-97f6-5271a1a9d3bc") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:13.610943 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:13.610367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:13.610943 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.610528    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:11:13.610943 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.610547    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:11:13.610943 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.610561    2573 projected.go:196] Error preparing data for projected volume kube-api-access-67hfl for pod openshift-network-diagnostics/network-check-target-qlmln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:13.610943 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.610618    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl podName:c15a2707-8b56-4c41-8d37-98cd53e9d4e6 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:21.61059998 +0000 UTC m=+17.270397168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-67hfl" (UniqueName: "kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl") pod "network-check-target-qlmln" (UID: "c15a2707-8b56-4c41-8d37-98cd53e9d4e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:13.828199 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:13.828098    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:13.828342 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.828247    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:13.828594 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:13.828580    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:13.828685 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.828669    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:13.828733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:13.828718    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:13.828800 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:13.828770    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:15.827992 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:15.827418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:15.827992 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:15.827554    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:15.827992 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:15.827940    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:15.827992 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:15.827953    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:15.828528 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:15.828033    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:15.828528 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:15.828123    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:17.827117 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:17.827074    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:17.827567 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:17.827075    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:17.827567 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:17.827209    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:17.827567 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:17.827082    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:17.827567 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:17.827288    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:17.827567 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:17.827409    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:19.828037 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:19.828001    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:19.828037 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:19.828029    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:19.828524 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:19.828116    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:19.828524 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:19.828176    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:19.828524 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:19.828303    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:19.828524 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:19.828387    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:21.470966 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:21.470942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:21.471369 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.471093    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:21.471369 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.471162    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs podName:46be1c16-0b44-44ae-8675-69b7b8442231 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:37.471132609 +0000 UTC m=+33.130929805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs") pod "network-metrics-daemon-5v6bv" (UID: "46be1c16-0b44-44ae-8675-69b7b8442231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:21.571497 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:21.571463    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:21.571658 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.571599    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:21.571658 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.571655    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret podName:bfc622a4-f58d-42ec-97f6-5271a1a9d3bc nodeName:}" failed. No retries permitted until 2026-09-10 19:11:37.571641751 +0000 UTC m=+33.231438922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret") pod "global-pull-secret-syncer-htz6q" (UID: "bfc622a4-f58d-42ec-97f6-5271a1a9d3bc") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:21.672703 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:21.672661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:21.672869 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.672821    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:11:21.672869 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.672840    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:11:21.672869 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.672849    2573 projected.go:196] Error preparing data for projected volume kube-api-access-67hfl for pod openshift-network-diagnostics/network-check-target-qlmln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:21.673035 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.672898    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl podName:c15a2707-8b56-4c41-8d37-98cd53e9d4e6 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:37.672885351 +0000 UTC m=+33.332682521 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-67hfl" (UniqueName: "kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl") pod "network-check-target-qlmln" (UID: "c15a2707-8b56-4c41-8d37-98cd53e9d4e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:21.828061 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:21.828008    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:21.828282 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:21.828129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:21.828282 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:21.828165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:21.828282 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.828140    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:21.828282 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.828227    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:21.828452 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:21.828294    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:23.827613 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:23.827574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:23.828170 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:23.827627    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:23.828170 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:23.827684    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:23.828170 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:23.827703    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:23.828170 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:23.827786    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:23.828170 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:23.827847    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:25.828099 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:25.827920    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:25.828718 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:25.827960    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:25.828718 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:25.828215    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:25.828718 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:25.827960    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:25.828718 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:25.828286    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:25.828718 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:25.828323    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:26.006872 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:26.006833    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cxb2v" event={"ID":"dbf8a58b-9706-4782-acc4-69500bedc6fc","Type":"ContainerStarted","Data":"d6b9990f9b3a07f43870af5c6489e06e9b34763ebd634858e353e970c06dee6f"}
Sep 10 19:11:26.027784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:26.027752    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" event={"ID":"6cbe649ca88acb6ca8918c435c58b662","Type":"ContainerStarted","Data":"0f18fb436e5c04c09cb2aebe090dad469327fb843d1a90419a39190c7ec02128"}
Sep 10 19:11:26.033378 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:26.033318    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cxb2v" podStartSLOduration=1.6700370150000001 podStartE2EDuration="21.033298247s" totalImagesPullingTime="19.363261232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:26.032746413 +0000 UTC m=+21.692543607" watchObservedRunningTime="2026-09-10 19:11:26.033298247 +0000 UTC m=+21.693095443"
Sep 10 19:11:26.037450 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:26.037417    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5rfzq" event={"ID":"0aa983d7-eafe-4eb0-9cf1-b236e928a649","Type":"ContainerStarted","Data":"b55f7604aea2f23cdc321e485b69cdf9ec9b73478a8958601a2304eecefc539e"}
Sep 10 19:11:26.049190 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:26.049102    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-45.ec2.internal" podStartSLOduration=21.049080357 podStartE2EDuration="21.049080357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:26.048035875 +0000 UTC m=+21.707833070" watchObservedRunningTime="2026-09-10 19:11:26.049080357 +0000 UTC m=+21.708877551"
Sep 10 19:11:26.065762 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:26.065702    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5rfzq" podStartSLOduration=1.407993724 podStartE2EDuration="21.065680084s" totalImagesPullingTime="19.65768636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:26.065176753 +0000 UTC m=+21.724973948" watchObservedRunningTime="2026-09-10 19:11:26.065680084 +0000 UTC m=+21.725477278"
Sep 10 19:11:27.041608 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.041395    2573 generic.go:356] "Generic (PLEG): container finished" podID="72f6adf29d1c71bfdc358a4f97e0872d" containerID="f6a5cb933db83f80062c54718de5674cd94dfb3838fbabd93604d5c81a3aa9d7" exitCode=0
Sep 10 19:11:27.042409 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.041600    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"72f6adf29d1c71bfdc358a4f97e0872d","Type":"ContainerDied","Data":"f6a5cb933db83f80062c54718de5674cd94dfb3838fbabd93604d5c81a3aa9d7"}
Sep 10 19:11:27.043521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.043490    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wn7z8" event={"ID":"356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b","Type":"ContainerStarted","Data":"da665d7afa09c957b237174fb224833d0d453f50180c7548bf1f43a83b354954"}
Sep 10 19:11:27.045322 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.045299    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv" event={"ID":"22c1eacb-21a4-45db-ba0a-5e727cb382e3","Type":"ContainerStarted","Data":"aa8fe53f16594836c7a036b1e13eaaca34bbb77c2f800afb6274b9c6f3bb2e7e"}
Sep 10 19:11:27.047163 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.047119    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctj27" event={"ID":"4d5123b1-49ac-4bcc-99d3-ef530a387f86","Type":"ContainerStarted","Data":"a6857545dd83c7de00923622c36d71ecf8fb82083c016486bec6451060361d4a"}
Sep 10 19:11:27.053428 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.053400    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" event={"ID":"0b8ab082-b524-4dc7-b4a2-73f54fb82423","Type":"ContainerStarted","Data":"79fdf51227cde553e8dd627658b516cc918f5841d5ca79aa63934461251e53c4"}
Sep 10 19:11:27.053526 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.053486    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" event={"ID":"0b8ab082-b524-4dc7-b4a2-73f54fb82423","Type":"ContainerStarted","Data":"c954743c6574668dcef61fe50d78e2b290ebde9d3375d07033f83fd6b6efc07f"}
Sep 10 19:11:27.053583 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.053542    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" event={"ID":"0b8ab082-b524-4dc7-b4a2-73f54fb82423","Type":"ContainerStarted","Data":"f957f20460b6e69dced6cbe97b054931b934dc5f97bbab149b7f6fcacec0f504"}
Sep 10 19:11:27.053637 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.053592    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" event={"ID":"0b8ab082-b524-4dc7-b4a2-73f54fb82423","Type":"ContainerStarted","Data":"06a1255482281165e1c92c1224b542b4a0013e6ccc5acd7859bd356122b22065"}
Sep 10 19:11:27.053690 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.053638    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" event={"ID":"0b8ab082-b524-4dc7-b4a2-73f54fb82423","Type":"ContainerStarted","Data":"6d8cc6dd5d63c0ea266fa49b0a63239c15b9341da3a413f3d1d0f4767f319d73"}
Sep 10 19:11:27.053749 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.053689    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" event={"ID":"0b8ab082-b524-4dc7-b4a2-73f54fb82423","Type":"ContainerStarted","Data":"a4acbe65c7fbd19939e9e620b534a71346c3f45d36aaeaceaa4bf51d4776b2ae"}
Sep 10 19:11:27.055133 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.055108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mbsgt" event={"ID":"cc8c22ae-6623-4996-a91f-1007228beb5e","Type":"ContainerStarted","Data":"a4c8526137d070a12966c7bd49737088f09de54c44e6152aaceacfe2f0c0ce3e"}
Sep 10 19:11:27.056936 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.056914    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f1754fb-fef4-4250-bffa-2032c9acdd11" containerID="9405d8816b606188a9f39508abdebbb8ece10d70f7ad2647b5d5b4bf16e485cb" exitCode=0
Sep 10 19:11:27.057027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.056962    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-967gq" event={"ID":"0f1754fb-fef4-4250-bffa-2032c9acdd11","Type":"ContainerDied","Data":"9405d8816b606188a9f39508abdebbb8ece10d70f7ad2647b5d5b4bf16e485cb"}
Sep 10 19:11:27.096383 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.096327    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wn7z8" podStartSLOduration=2.6416283419999997 podStartE2EDuration="22.09630477s" totalImagesPullingTime="19.454676428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:27.095801389 +0000 UTC m=+22.755598582" watchObservedRunningTime="2026-09-10 19:11:27.09630477 +0000 UTC m=+22.756101979"
Sep 10 19:11:27.125993 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.125942    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ctj27" podStartSLOduration=2.769591435 podStartE2EDuration="22.125925231s" totalImagesPullingTime="19.356333796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:27.110555946 +0000 UTC m=+22.770353140" watchObservedRunningTime="2026-09-10 19:11:27.125925231 +0000 UTC m=+22.785722496"
Sep 10 19:11:27.126408 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.126368    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mbsgt" podStartSLOduration=2.721621943 podStartE2EDuration="22.126354595s" totalImagesPullingTime="19.404732652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:27.125802528 +0000 UTC m=+22.785599724" watchObservedRunningTime="2026-09-10 19:11:27.126354595 +0000 UTC m=+22.786151795"
Sep 10 19:11:27.827610 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.827578    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:27.827778 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:27.827714    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:27.828170 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.828129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:27.828276 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:27.828252    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:27.828339 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.828320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:27.828437 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:27.828411    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:27.925760 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:27.925731    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 19:11:28.061620 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.061525    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" event={"ID":"72f6adf29d1c71bfdc358a4f97e0872d","Type":"ContainerStarted","Data":"efbd511bb68fc84729ac100f36bcf7c0e9fae9880d8f49ed6628e224f4b2990d"}
Sep 10 19:11:28.063093 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.063067    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7r4lv" event={"ID":"f1c4d93c-dd02-4ea0-b085-97afdf95ac47","Type":"ContainerStarted","Data":"95c3e5b029ce205e590f7055e9ddceda97ecf3d3b45878e1e833e162396ab3cf"}
Sep 10 19:11:28.064998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.064975    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv" event={"ID":"22c1eacb-21a4-45db-ba0a-5e727cb382e3","Type":"ContainerStarted","Data":"1694c195faff08138e9f0938deb1def8103a8dac3e473875338f910a8de750d9"}
Sep 10 19:11:28.081225 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.081169    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-45.ec2.internal" podStartSLOduration=23.081138602 podStartE2EDuration="23.081138602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:28.081127224 +0000 UTC m=+23.740924416" watchObservedRunningTime="2026-09-10 19:11:28.081138602 +0000 UTC m=+23.740935797"
Sep 10 19:11:28.101050 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.100999    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7r4lv" podStartSLOduration=3.678273386 podStartE2EDuration="23.100976556s" totalImagesPullingTime="19.42270317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:28.100804852 +0000 UTC m=+23.760602056" watchObservedRunningTime="2026-09-10 19:11:28.100976556 +0000 UTC m=+23.760773754"
Sep 10 19:11:28.196831 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.196744    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:28.197436 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.197416    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:28.799481 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.799183    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T19:11:27.92575202Z","UUID":"8ae5038e-0086-4653-804d-e4bbe2526b9d","Handler":null,"Name":"","Endpoint":""}
Sep 10 19:11:28.800916 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.800893    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 19:11:28.801046 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:28.800934    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 19:11:29.071907 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:29.071868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" event={"ID":"0b8ab082-b524-4dc7-b4a2-73f54fb82423","Type":"ContainerStarted","Data":"3277516edd1b987e216c5f9a2a95276255750066b3f99b71eaa3e9dd6609bc85"}
Sep 10 19:11:29.827880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:29.827846    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:29.828081 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:29.827887    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:29.828081 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:29.827887    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:29.828081 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:29.827978    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:29.828269 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:29.828077    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:29.828269 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:29.828199    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:30.076192 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:30.076144    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 19:11:30.076192 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:30.076179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv" event={"ID":"22c1eacb-21a4-45db-ba0a-5e727cb382e3","Type":"ContainerStarted","Data":"c0e7147f8bae779adcf94fd960b3bf532d166d6b1dfc435d295d01a04c1de8d2"}
Sep 10 19:11:30.097840 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:30.097746    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x87tv" podStartSLOduration=3.853139567 podStartE2EDuration="26.097730332s" totalImagesPullingTime="22.244590765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:30.09743016 +0000 UTC m=+25.757227353" watchObservedRunningTime="2026-09-10 19:11:30.097730332 +0000 UTC m=+25.757527530"
Sep 10 19:11:31.082899 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:31.082856    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" event={"ID":"0b8ab082-b524-4dc7-b4a2-73f54fb82423","Type":"ContainerStarted","Data":"7f062fa0f240e1eaa25e324f2aee31ee6150e810e0d6822fa8862e3ead904e63"}
Sep 10 19:11:31.828103 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:31.828054    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:31.828103 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:31.828085    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:31.828331 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:31.828208    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:31.828331 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:31.828258    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:31.828415 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:31.828341    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:31.828474 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:31.828440    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:32.085808 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:32.085714    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:32.085808 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:32.085806    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:32.086282 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:32.085858    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:32.103410 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:32.103376    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:32.103564 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:32.103519    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:11:32.143954 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:32.143902    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg" podStartSLOduration=7.3809495290000005 podStartE2EDuration="27.143882543s" totalImagesPullingTime="19.762933014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:31.128430713 +0000 UTC m=+26.788227906" watchObservedRunningTime="2026-09-10 19:11:32.143882543 +0000 UTC m=+27.803679736"
Sep 10 19:11:33.089099 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:33.088875    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f1754fb-fef4-4250-bffa-2032c9acdd11" containerID="c18754251ee04ed43a01d45cbe8cacc143e91c726a725930c2b5682411d12898" exitCode=0
Sep 10 19:11:33.089500 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:33.088962    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-967gq" event={"ID":"0f1754fb-fef4-4250-bffa-2032c9acdd11","Type":"ContainerDied","Data":"c18754251ee04ed43a01d45cbe8cacc143e91c726a725930c2b5682411d12898"}
Sep 10 19:11:33.827644 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:33.827606    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:33.827935 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:33.827654    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:33.827935 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:33.827708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:33.827935 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:33.827721    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:33.827935 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:33.827798    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:33.827935 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:33.827865    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:34.957265 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:34.957189    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:34.957633 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:34.957329    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 19:11:34.957860 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:34.957840    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wn7z8"
Sep 10 19:11:35.098585 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:35.098550    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f1754fb-fef4-4250-bffa-2032c9acdd11" containerID="bdf0c9f9d1e99f43a5c28d6956ded33419d5665faa412bd76f024760b7afd15f" exitCode=0
Sep 10 19:11:35.098748 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:35.098645    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-967gq" event={"ID":"0f1754fb-fef4-4250-bffa-2032c9acdd11","Type":"ContainerDied","Data":"bdf0c9f9d1e99f43a5c28d6956ded33419d5665faa412bd76f024760b7afd15f"}
Sep 10 19:11:35.828056 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:35.828017    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:35.828056 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:35.828044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:35.828339 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:35.828017    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:35.828339 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:35.828165    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:35.828339 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:35.828235    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:35.828339 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:35.828306    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:37.106597 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:37.106563    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f1754fb-fef4-4250-bffa-2032c9acdd11" containerID="1c3fef1f74f8da69096f2d0d0eda8ac19defceb4c78bbeb093c51e32056e6190" exitCode=0
Sep 10 19:11:37.107016 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:37.106615    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-967gq" event={"ID":"0f1754fb-fef4-4250-bffa-2032c9acdd11","Type":"ContainerDied","Data":"1c3fef1f74f8da69096f2d0d0eda8ac19defceb4c78bbeb093c51e32056e6190"}
Sep 10 19:11:37.488213 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:37.488099    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:37.488373 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.488260    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:37.488373 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.488322    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs podName:46be1c16-0b44-44ae-8675-69b7b8442231 nodeName:}" failed. No retries permitted until 2026-09-10 19:12:09.488307895 +0000 UTC m=+65.148105080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs") pod "network-metrics-daemon-5v6bv" (UID: "46be1c16-0b44-44ae-8675-69b7b8442231") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:37.589317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:37.589275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:37.589484 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.589400    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:37.589484 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.589453    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret podName:bfc622a4-f58d-42ec-97f6-5271a1a9d3bc nodeName:}" failed. No retries permitted until 2026-09-10 19:12:09.58944059 +0000 UTC m=+65.249237761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret") pod "global-pull-secret-syncer-htz6q" (UID: "bfc622a4-f58d-42ec-97f6-5271a1a9d3bc") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:37.689728 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:37.689686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:37.689903 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.689835    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:11:37.689903 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.689855    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:11:37.689903 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.689868    2573 projected.go:196] Error preparing data for projected volume kube-api-access-67hfl for pod openshift-network-diagnostics/network-check-target-qlmln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:37.690055 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.689932    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl podName:c15a2707-8b56-4c41-8d37-98cd53e9d4e6 nodeName:}" failed. No retries permitted until 2026-09-10 19:12:09.689915376 +0000 UTC m=+65.349712573 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-67hfl" (UniqueName: "kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl") pod "network-check-target-qlmln" (UID: "c15a2707-8b56-4c41-8d37-98cd53e9d4e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:37.827180 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:37.827134    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:37.827180 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:37.827172    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:37.827443 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:37.827273    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:37.827443 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.827295    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:37.827443 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.827397    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:37.827587 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:37.827464    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:39.827337 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:39.827301    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:39.827337 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:39.827343    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:39.828103 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:39.827308    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:39.828103 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:39.827436    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:39.828103 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:39.827542    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:39.828103 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:39.827579    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:41.827781 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:41.827742    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:41.827781 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:41.827780    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:41.828318 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:41.827853    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:41.828318 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:41.827878    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:41.828318 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:41.827928    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:41.828318 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:41.827982    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:43.124101 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:43.124066    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f1754fb-fef4-4250-bffa-2032c9acdd11" containerID="617e7a6a52b570f2fc4128f63b78dd6d0e671ba9987dafca68b333cda0c68da1" exitCode=0
Sep 10 19:11:43.124476 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:43.124122    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-967gq" event={"ID":"0f1754fb-fef4-4250-bffa-2032c9acdd11","Type":"ContainerDied","Data":"617e7a6a52b570f2fc4128f63b78dd6d0e671ba9987dafca68b333cda0c68da1"}
Sep 10 19:11:43.827669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:43.827625    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:43.827871 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:43.827673    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:43.827871 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:43.827732    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:43.827871 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:43.827743    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:43.827871 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:43.827829    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:43.828043 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:43.827910    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:44.129762 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:44.129677    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f1754fb-fef4-4250-bffa-2032c9acdd11" containerID="0c8e1a7b1d43df2c4e4b9834a1db6268e3ff2f02869a0e497f1736fe63741f20" exitCode=0
Sep 10 19:11:44.129762 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:44.129723    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-967gq" event={"ID":"0f1754fb-fef4-4250-bffa-2032c9acdd11","Type":"ContainerDied","Data":"0c8e1a7b1d43df2c4e4b9834a1db6268e3ff2f02869a0e497f1736fe63741f20"}
Sep 10 19:11:45.138521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:45.138316    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-967gq" event={"ID":"0f1754fb-fef4-4250-bffa-2032c9acdd11","Type":"ContainerStarted","Data":"e81b4027338715a1d02cb0e3dace0e244bd828df663f45bdfe766f886bf0042e"}
Sep 10 19:11:45.160757 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:45.160712    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-967gq" podStartSLOduration=7.053855526 podStartE2EDuration="40.160698175s" totalImagesPullingTime="33.106842649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:45.15918602 +0000 UTC m=+40.818983213" watchObservedRunningTime="2026-09-10 19:11:45.160698175 +0000 UTC m=+40.820495379"
Sep 10 19:11:45.827818 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:45.827787    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:45.828001 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:45.827873    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:45.828001 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:45.827879    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:45.828001 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:45.827954    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:45.828001 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:45.827960    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:45.828189 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:45.828036    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:47.772350 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:47.772287    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5v6bv"]
Sep 10 19:11:47.773039 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:47.772450    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:47.773039 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:47.772550    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:47.775377 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:47.775310    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-htz6q"]
Sep 10 19:11:47.775514 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:47.775420    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:47.775609 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:47.775521    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:47.775879 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:47.775857    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qlmln"]
Sep 10 19:11:47.775975 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:47.775944    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:47.776116 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:47.776034    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:49.827661 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:49.827616    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:49.828058 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:49.827616    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:49.828058 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:49.827740    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5v6bv" podUID="46be1c16-0b44-44ae-8675-69b7b8442231"
Sep 10 19:11:49.828058 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:49.827616    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:49.828058 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:49.827811    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlmln" podUID="c15a2707-8b56-4c41-8d37-98cd53e9d4e6"
Sep 10 19:11:49.828058 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:11:49.827874    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-htz6q" podUID="bfc622a4-f58d-42ec-97f6-5271a1a9d3bc"
Sep 10 19:11:51.618497 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.618420    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-45.ec2.internal" event="NodeReady"
Sep 10 19:11:51.618908 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.618538    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 19:11:51.673317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.673290    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-97hkg"]
Sep 10 19:11:51.689810 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.689781    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-77nq5"]
Sep 10 19:11:51.689943 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.689921    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-97hkg"
Sep 10 19:11:51.692556 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.692527    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 19:11:51.692684 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.692554    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 19:11:51.692684 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.692555    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 19:11:51.692684 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.692617    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zn8jq\""
Sep 10 19:11:51.703868 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.703849    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-97hkg"]
Sep 10 19:11:51.703961 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.703876    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-77nq5"]
Sep 10 19:11:51.703999 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.703964    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.706687 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.706671    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 19:11:51.706901 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.706878    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 19:11:51.707063 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.707043    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 19:11:51.707323 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.707308    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 19:11:51.707396 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.707322    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zl4hx\""
Sep 10 19:11:51.766847 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.766818    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b8fgm"]
Sep 10 19:11:51.780626 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.780594    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljpfg\" (UniqueName: \"kubernetes.io/projected/f1110bb8-b19b-49cc-877c-dc28e872558f-kube-api-access-ljpfg\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.780782 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.780630    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5d46\" (UniqueName: \"kubernetes.io/projected/b1560abb-47cd-4eff-ac93-e3f9f1deb86f-kube-api-access-t5d46\") pod \"ingress-canary-97hkg\" (UID: \"b1560abb-47cd-4eff-ac93-e3f9f1deb86f\") " pod="openshift-ingress-canary/ingress-canary-97hkg"
Sep 10 19:11:51.780782 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.780657    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1110bb8-b19b-49cc-877c-dc28e872558f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.780782 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.780723    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1560abb-47cd-4eff-ac93-e3f9f1deb86f-cert\") pod \"ingress-canary-97hkg\" (UID: \"b1560abb-47cd-4eff-ac93-e3f9f1deb86f\") " pod="openshift-ingress-canary/ingress-canary-97hkg"
Sep 10 19:11:51.780782 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.780753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1110bb8-b19b-49cc-877c-dc28e872558f-crio-socket\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.780937 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.780799    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1110bb8-b19b-49cc-877c-dc28e872558f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.780937 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.780884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1110bb8-b19b-49cc-877c-dc28e872558f-data-volume\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.788634 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.788607    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8fgm"]
Sep 10 19:11:51.788777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.788738    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.791296 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.791275    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b59cd\""
Sep 10 19:11:51.791408 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.791275    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 19:11:51.791408 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.791276    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 19:11:51.827787 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.827751    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:11:51.827955 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.827843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:11:51.827955 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.827868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:11:51.830545 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.830520    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zj5gq\""
Sep 10 19:11:51.830672 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.830627    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 19:11:51.830672 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.830621    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 19:11:51.830789 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.830748    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8gdr\""
Sep 10 19:11:51.830844 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.830790    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 19:11:51.830844 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.830749    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 19:11:51.882040 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.881952    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1110bb8-b19b-49cc-877c-dc28e872558f-data-volume\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.882040 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.881989    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljpfg\" (UniqueName: \"kubernetes.io/projected/f1110bb8-b19b-49cc-877c-dc28e872558f-kube-api-access-ljpfg\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.882040 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882010    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zq6n\" (UniqueName: \"kubernetes.io/projected/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-kube-api-access-8zq6n\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.882040 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882028    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5d46\" (UniqueName: \"kubernetes.io/projected/b1560abb-47cd-4eff-ac93-e3f9f1deb86f-kube-api-access-t5d46\") pod \"ingress-canary-97hkg\" (UID: \"b1560abb-47cd-4eff-ac93-e3f9f1deb86f\") " pod="openshift-ingress-canary/ingress-canary-97hkg"
Sep 10 19:11:51.882040 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1110bb8-b19b-49cc-877c-dc28e872558f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.882415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882064    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-metrics-tls\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.882415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-config-volume\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.882415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882189    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1560abb-47cd-4eff-ac93-e3f9f1deb86f-cert\") pod \"ingress-canary-97hkg\" (UID: \"b1560abb-47cd-4eff-ac93-e3f9f1deb86f\") " pod="openshift-ingress-canary/ingress-canary-97hkg"
Sep 10 19:11:51.882415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1110bb8-b19b-49cc-877c-dc28e872558f-crio-socket\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.882415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1110bb8-b19b-49cc-877c-dc28e872558f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.882415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882404    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-tmp-dir\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.882617 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882436    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f1110bb8-b19b-49cc-877c-dc28e872558f-crio-socket\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.882617 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882371    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f1110bb8-b19b-49cc-877c-dc28e872558f-data-volume\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.882835 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.882818    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f1110bb8-b19b-49cc-877c-dc28e872558f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.886509 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.886405    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f1110bb8-b19b-49cc-877c-dc28e872558f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.886764 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.886742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1560abb-47cd-4eff-ac93-e3f9f1deb86f-cert\") pod \"ingress-canary-97hkg\" (UID: \"b1560abb-47cd-4eff-ac93-e3f9f1deb86f\") " pod="openshift-ingress-canary/ingress-canary-97hkg"
Sep 10 19:11:51.892085 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.892063    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljpfg\" (UniqueName: \"kubernetes.io/projected/f1110bb8-b19b-49cc-877c-dc28e872558f-kube-api-access-ljpfg\") pod \"insights-runtime-extractor-77nq5\" (UID: \"f1110bb8-b19b-49cc-877c-dc28e872558f\") " pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:51.892085 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.892083    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5d46\" (UniqueName: \"kubernetes.io/projected/b1560abb-47cd-4eff-ac93-e3f9f1deb86f-kube-api-access-t5d46\") pod \"ingress-canary-97hkg\" (UID: \"b1560abb-47cd-4eff-ac93-e3f9f1deb86f\") " pod="openshift-ingress-canary/ingress-canary-97hkg"
Sep 10 19:11:51.983295 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.983258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zq6n\" (UniqueName: \"kubernetes.io/projected/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-kube-api-access-8zq6n\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.983295 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.983299    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-metrics-tls\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.983551 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.983327    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-config-volume\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.983551 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.983361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-tmp-dir\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.983784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.983763    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-tmp-dir\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.983957 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.983937    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-config-volume\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.985798 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.985772    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-metrics-tls\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:51.992204 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:51.992183    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zq6n\" (UniqueName: \"kubernetes.io/projected/8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c-kube-api-access-8zq6n\") pod \"dns-default-b8fgm\" (UID: \"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c\") " pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:52.000099 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:52.000062    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-97hkg"
Sep 10 19:11:52.013315 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:52.013295    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-77nq5"
Sep 10 19:11:52.097355 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:52.097320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:52.329227 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:52.329190    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-97hkg"]
Sep 10 19:11:52.333165 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:52.333122    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-77nq5"]
Sep 10 19:11:52.333520 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:52.333494    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1560abb_47cd_4eff_ac93_e3f9f1deb86f.slice/crio-384df07e5123ea71da7c798a1b809ad73b4d97f88927f5850899f635adf202ac WatchSource:0}: Error finding container 384df07e5123ea71da7c798a1b809ad73b4d97f88927f5850899f635adf202ac: Status 404 returned error can't find the container with id 384df07e5123ea71da7c798a1b809ad73b4d97f88927f5850899f635adf202ac
Sep 10 19:11:52.337591 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:52.337287    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1110bb8_b19b_49cc_877c_dc28e872558f.slice/crio-44c95931d698e645e04f8e6fe87b9f39d338d9ee013176305ae17795808b4892 WatchSource:0}: Error finding container 44c95931d698e645e04f8e6fe87b9f39d338d9ee013176305ae17795808b4892: Status 404 returned error can't find the container with id 44c95931d698e645e04f8e6fe87b9f39d338d9ee013176305ae17795808b4892
Sep 10 19:11:52.393501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:52.393406    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b8fgm"]
Sep 10 19:11:52.397290 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:52.397248    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5678b7_5ab3_454f_ae3a_8b1eb6d6d72c.slice/crio-005207e915898c83782966cddb77be3d5335401ccd3574a7d5994adaf681fb52 WatchSource:0}: Error finding container 005207e915898c83782966cddb77be3d5335401ccd3574a7d5994adaf681fb52: Status 404 returned error can't find the container with id 005207e915898c83782966cddb77be3d5335401ccd3574a7d5994adaf681fb52
Sep 10 19:11:53.159198 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:53.159161    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77nq5" event={"ID":"f1110bb8-b19b-49cc-877c-dc28e872558f","Type":"ContainerStarted","Data":"78d7543a0b7219974fe28cd874ead378ff0c7a5dce26a7d034f06ca187823c4a"}
Sep 10 19:11:53.160237 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:53.159236    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77nq5" event={"ID":"f1110bb8-b19b-49cc-877c-dc28e872558f","Type":"ContainerStarted","Data":"44c95931d698e645e04f8e6fe87b9f39d338d9ee013176305ae17795808b4892"}
Sep 10 19:11:53.160901 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:53.160869    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-97hkg" event={"ID":"b1560abb-47cd-4eff-ac93-e3f9f1deb86f","Type":"ContainerStarted","Data":"384df07e5123ea71da7c798a1b809ad73b4d97f88927f5850899f635adf202ac"}
Sep 10 19:11:53.162373 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:53.162320    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8fgm" event={"ID":"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c","Type":"ContainerStarted","Data":"005207e915898c83782966cddb77be3d5335401ccd3574a7d5994adaf681fb52"}
Sep 10 19:11:54.167413 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.167367    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77nq5" event={"ID":"f1110bb8-b19b-49cc-877c-dc28e872558f","Type":"ContainerStarted","Data":"dcd1b22af15a316a896a0b5ae3badcf47fd55d8abe187a76d6d365340d30dc86"}
Sep 10 19:11:54.352356 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.352315    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-6vdrv"]
Sep 10 19:11:54.355591 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.355532    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6vdrv"
Sep 10 19:11:54.358252 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.358224    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 19:11:54.358361 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.358260    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 19:11:54.358567 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.358551    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fwnps\""
Sep 10 19:11:54.368448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.368175    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6vdrv"]
Sep 10 19:11:54.504607 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.504450    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbq9z\" (UniqueName: \"kubernetes.io/projected/41c2f980-3cbf-4fe6-8daf-d90a57fec390-kube-api-access-mbq9z\") pod \"downloads-85d98c998f-6vdrv\" (UID: \"41c2f980-3cbf-4fe6-8daf-d90a57fec390\") " pod="openshift-console/downloads-85d98c998f-6vdrv"
Sep 10 19:11:54.605638 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.605592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbq9z\" (UniqueName: \"kubernetes.io/projected/41c2f980-3cbf-4fe6-8daf-d90a57fec390-kube-api-access-mbq9z\") pod \"downloads-85d98c998f-6vdrv\" (UID: \"41c2f980-3cbf-4fe6-8daf-d90a57fec390\") " pod="openshift-console/downloads-85d98c998f-6vdrv"
Sep 10 19:11:54.615314 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.615281    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbq9z\" (UniqueName: \"kubernetes.io/projected/41c2f980-3cbf-4fe6-8daf-d90a57fec390-kube-api-access-mbq9z\") pod \"downloads-85d98c998f-6vdrv\" (UID: \"41c2f980-3cbf-4fe6-8daf-d90a57fec390\") " pod="openshift-console/downloads-85d98c998f-6vdrv"
Sep 10 19:11:54.669485 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.669441    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-6vdrv"
Sep 10 19:11:54.841894 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.841857    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"]
Sep 10 19:11:54.845438 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.845409    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:54.848110 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.847924    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 19:11:54.848282 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.847954    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 19:11:54.848363 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.848000    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 19:11:54.848414 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.848005    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mrrlx\""
Sep 10 19:11:54.848470 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.848052    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 19:11:54.848520 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.848080    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 19:11:54.858284 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:54.858262    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"]
Sep 10 19:11:55.009396 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.009354    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b879388-e731-454a-a839-b5ae3572b3c0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.009558 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.009415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b879388-e731-454a-a839-b5ae3572b3c0-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.009558 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.009462    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b879388-e731-454a-a839-b5ae3572b3c0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.009558 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.009538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7cv\" (UniqueName: \"kubernetes.io/projected/6b879388-e731-454a-a839-b5ae3572b3c0-kube-api-access-7j7cv\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.110880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.110772    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b879388-e731-454a-a839-b5ae3572b3c0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.110880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.110823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b879388-e731-454a-a839-b5ae3572b3c0-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.110880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.110867    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b879388-e731-454a-a839-b5ae3572b3c0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.111130 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.110913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7cv\" (UniqueName: \"kubernetes.io/projected/6b879388-e731-454a-a839-b5ae3572b3c0-kube-api-access-7j7cv\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.111716 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.111686    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b879388-e731-454a-a839-b5ae3572b3c0-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.113576 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.113552    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b879388-e731-454a-a839-b5ae3572b3c0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.114034 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.114010    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b879388-e731-454a-a839-b5ae3572b3c0-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.120261 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.120226    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7cv\" (UniqueName: \"kubernetes.io/projected/6b879388-e731-454a-a839-b5ae3572b3c0-kube-api-access-7j7cv\") pod \"prometheus-operator-6f4946db56-x2gc4\" (UID: \"6b879388-e731-454a-a839-b5ae3572b3c0\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.156419 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.156384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"
Sep 10 19:11:55.731004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.730920    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-6vdrv"]
Sep 10 19:11:55.733558 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:55.733534    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-x2gc4"]
Sep 10 19:11:55.736885 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:55.736858    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c2f980_3cbf_4fe6_8daf_d90a57fec390.slice/crio-0ec8127aaa114f9afb150ba236b5d22623f999338ee42ce751fac2ebd9cf6a1d WatchSource:0}: Error finding container 0ec8127aaa114f9afb150ba236b5d22623f999338ee42ce751fac2ebd9cf6a1d: Status 404 returned error can't find the container with id 0ec8127aaa114f9afb150ba236b5d22623f999338ee42ce751fac2ebd9cf6a1d
Sep 10 19:11:55.737344 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:11:55.737320    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b879388_e731_454a_a839_b5ae3572b3c0.slice/crio-869254033b6c52da22ddcd9a69c8751bec0310fa3d22dd435c250f91381f8d1f WatchSource:0}: Error finding container 869254033b6c52da22ddcd9a69c8751bec0310fa3d22dd435c250f91381f8d1f: Status 404 returned error can't find the container with id 869254033b6c52da22ddcd9a69c8751bec0310fa3d22dd435c250f91381f8d1f
Sep 10 19:11:56.176328 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.176288    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-97hkg" event={"ID":"b1560abb-47cd-4eff-ac93-e3f9f1deb86f","Type":"ContainerStarted","Data":"42822f1ea6fea85133579da5b53cf0167d62280c76b71e69fcbf9d29e033ae4e"}
Sep 10 19:11:56.178093 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.178056    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8fgm" event={"ID":"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c","Type":"ContainerStarted","Data":"b5e728a22b38fb59ae76226b5fb0227a50eccd0d4a294849bf9beedd6bb9ca2d"}
Sep 10 19:11:56.178246 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.178110    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b8fgm" event={"ID":"8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c","Type":"ContainerStarted","Data":"c74b913a27b3e06ac1b9f4b1505b3dfde7861ef15f3f6a8ac23aae9bc0ba9bdd"}
Sep 10 19:11:56.178246 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.178139    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:11:56.179358 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.179281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4" event={"ID":"6b879388-e731-454a-a839-b5ae3572b3c0","Type":"ContainerStarted","Data":"869254033b6c52da22ddcd9a69c8751bec0310fa3d22dd435c250f91381f8d1f"}
Sep 10 19:11:56.181347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.181327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77nq5" event={"ID":"f1110bb8-b19b-49cc-877c-dc28e872558f","Type":"ContainerStarted","Data":"32d3a8ebf65194a5dc1509e26d9805bc4dfee7269731cea45616ac1aeaac37bc"}
Sep 10 19:11:56.182422 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.182401    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6vdrv" event={"ID":"41c2f980-3cbf-4fe6-8daf-d90a57fec390","Type":"ContainerStarted","Data":"0ec8127aaa114f9afb150ba236b5d22623f999338ee42ce751fac2ebd9cf6a1d"}
Sep 10 19:11:56.194319 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.194280    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-97hkg" podStartSLOduration=2.069954989 podStartE2EDuration="5.194265678s" totalImagesPullingTime="3.124310689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:56.194044978 +0000 UTC m=+51.853842173" watchObservedRunningTime="2026-09-10 19:11:56.194265678 +0000 UTC m=+51.854062901"
Sep 10 19:11:56.223351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.223291    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-b8fgm" podStartSLOduration=2.171020111 podStartE2EDuration="5.223270916s" totalImagesPullingTime="3.052250805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:56.222291412 +0000 UTC m=+51.882088605" watchObservedRunningTime="2026-09-10 19:11:56.223270916 +0000 UTC m=+51.883068110"
Sep 10 19:11:56.240555 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:56.240505    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-77nq5" podStartSLOduration=2.291965179 podStartE2EDuration="5.240485919s" totalImagesPullingTime="2.94852074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:56.239955787 +0000 UTC m=+51.899752980" watchObservedRunningTime="2026-09-10 19:11:56.240485919 +0000 UTC m=+51.900283113"
Sep 10 19:11:57.192395 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:57.192355    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4" event={"ID":"6b879388-e731-454a-a839-b5ae3572b3c0","Type":"ContainerStarted","Data":"1184a35d133d4be5c135c0c965cd2fbe0166f8f04ef5b5a1e229e1be5aa50675"}
Sep 10 19:11:58.198843 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:58.198804    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4" event={"ID":"6b879388-e731-454a-a839-b5ae3572b3c0","Type":"ContainerStarted","Data":"cda9616d68f3dbfd8145aefd0fd8fdf244cb9f064fd3ab4f930e24445d810b02"}
Sep 10 19:11:58.221088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:11:58.221040    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-x2gc4" podStartSLOduration=2.855161354 podStartE2EDuration="4.221020819s" totalImagesPullingTime="1.365859465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:58.219500185 +0000 UTC m=+53.879297373" watchObservedRunningTime="2026-09-10 19:11:58.221020819 +0000 UTC m=+53.880818022"
Sep 10 19:12:00.210224 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.210194    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b7tbz"]
Sep 10 19:12:00.216095 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.216071    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"]
Sep 10 19:12:00.216682 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.216251    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.219028 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.219001    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 19:12:00.219193 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.219099    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 19:12:00.219342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.219323    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 19:12:00.219446 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.219379    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4fnns\""
Sep 10 19:12:00.220276 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.220259    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.222921 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.222901    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 19:12:00.223031 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.222939    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-khfcd\""
Sep 10 19:12:00.223088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.223055    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 19:12:00.243905 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.243791    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"]
Sep 10 19:12:00.246423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.245110    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"]
Sep 10 19:12:00.250110 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.249399    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"]
Sep 10 19:12:00.250110 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.249523    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.255694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.255663    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 19:12:00.255795 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.255729    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 19:12:00.255945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.255930    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-69lgr\""
Sep 10 19:12:00.256130 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.256114    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 19:12:00.352694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c542347-3526-4891-acd3-a052ded42597-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.352694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352708    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-accelerators-collector-config\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.352927 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c542347-3526-4891-acd3-a052ded42597-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.352927 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-tls\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.352927 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352772    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9f1b138-b565-4acd-a676-568aa20a8321-root\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.352927 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvnct\" (UniqueName: \"kubernetes.io/projected/a9f1b138-b565-4acd-a676-568aa20a8321-kube-api-access-rvnct\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.352927 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352872    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9f1b138-b565-4acd-a676-568aa20a8321-sys\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.352927 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352896    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.352927 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352922    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj45k\" (UniqueName: \"kubernetes.io/projected/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-api-access-mj45k\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352951    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.352997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.353021    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.353047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.353067    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.353121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-wtmp\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.353178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-textfile\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.353213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ddp7\" (UniqueName: \"kubernetes.io/projected/8c542347-3526-4891-acd3-a052ded42597-kube-api-access-7ddp7\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.353257    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c542347-3526-4891-acd3-a052ded42597-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.353291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.353291    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1b138-b565-4acd-a676-568aa20a8321-metrics-client-ca\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.453608 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.453576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c542347-3526-4891-acd3-a052ded42597-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.453758 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.453624    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1b138-b565-4acd-a676-568aa20a8321-metrics-client-ca\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.453758 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.453643    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c542347-3526-4891-acd3-a052ded42597-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.453758 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.453660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-accelerators-collector-config\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.453950 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.453924    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c542347-3526-4891-acd3-a052ded42597-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.453990 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.453972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-tls\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454040 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9f1b138-b565-4acd-a676-568aa20a8321-root\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454040 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454034    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvnct\" (UniqueName: \"kubernetes.io/projected/a9f1b138-b565-4acd-a676-568aa20a8321-kube-api-access-rvnct\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454132 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454066    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9f1b138-b565-4acd-a676-568aa20a8321-sys\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454132 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454089    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.454132 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454115    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj45k\" (UniqueName: \"kubernetes.io/projected/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-api-access-mj45k\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.454305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454164    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454220    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.454305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454246    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.454305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454279    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.454501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.454501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454319    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1b138-b565-4acd-a676-568aa20a8321-metrics-client-ca\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-wtmp\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-accelerators-collector-config\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9f1b138-b565-4acd-a676-568aa20a8321-root\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-textfile\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454471    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9f1b138-b565-4acd-a676-568aa20a8321-sys\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454487    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ddp7\" (UniqueName: \"kubernetes.io/projected/8c542347-3526-4891-acd3-a052ded42597-kube-api-access-7ddp7\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.454834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454526    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c542347-3526-4891-acd3-a052ded42597-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.454834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454577    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-wtmp\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.454834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.454834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.454802    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-textfile\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.455331 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.455285    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.455520 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.455475    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.458400 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.458375    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.458503 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.458420    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.458503 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.458425    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9f1b138-b565-4acd-a676-568aa20a8321-node-exporter-tls\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.458503 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.458463    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c542347-3526-4891-acd3-a052ded42597-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.458503 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.458496    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c542347-3526-4891-acd3-a052ded42597-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.458632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.458502    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.467587 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.467526    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvnct\" (UniqueName: \"kubernetes.io/projected/a9f1b138-b565-4acd-a676-568aa20a8321-kube-api-access-rvnct\") pod \"node-exporter-b7tbz\" (UID: \"a9f1b138-b565-4acd-a676-568aa20a8321\") " pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.468585 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.468559    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ddp7\" (UniqueName: \"kubernetes.io/projected/8c542347-3526-4891-acd3-a052ded42597-kube-api-access-7ddp7\") pod \"openshift-state-metrics-657fd6c54-2qnp5\" (UID: \"8c542347-3526-4891-acd3-a052ded42597\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.469695 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.469674    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj45k\" (UniqueName: \"kubernetes.io/projected/f5330f3a-ae25-4efa-92b6-fb2e6f0054ea-kube-api-access-mj45k\") pod \"kube-state-metrics-74cb57df7-fh2tx\" (UID: \"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.541460 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.541418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b7tbz"
Sep 10 19:12:00.555286 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:00.555250    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f1b138_b565_4acd_a676_568aa20a8321.slice/crio-c3d7b0f3e4c4d31fff7d0c34eeb256153d9cb64356b4661b7a09ab08182b86aa WatchSource:0}: Error finding container c3d7b0f3e4c4d31fff7d0c34eeb256153d9cb64356b4661b7a09ab08182b86aa: Status 404 returned error can't find the container with id c3d7b0f3e4c4d31fff7d0c34eeb256153d9cb64356b4661b7a09ab08182b86aa
Sep 10 19:12:00.558856 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.558832    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"
Sep 10 19:12:00.566203 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.566178    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"
Sep 10 19:12:00.844237 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.844206    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5"]
Sep 10 19:12:00.847912 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:00.847877    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c542347_3526_4891_acd3_a052ded42597.slice/crio-778855189ed0ec413984b944ed1428566b6b1f177ae487ba193081449fc50908 WatchSource:0}: Error finding container 778855189ed0ec413984b944ed1428566b6b1f177ae487ba193081449fc50908: Status 404 returned error can't find the container with id 778855189ed0ec413984b944ed1428566b6b1f177ae487ba193081449fc50908
Sep 10 19:12:00.855114 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:00.855092    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx"]
Sep 10 19:12:00.857269 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:00.857243    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5330f3a_ae25_4efa_92b6_fb2e6f0054ea.slice/crio-907f57a80a2e9a1e187639df1bb0846d2a3bedd56604be906b66dd60714382cc WatchSource:0}: Error finding container 907f57a80a2e9a1e187639df1bb0846d2a3bedd56604be906b66dd60714382cc: Status 404 returned error can't find the container with id 907f57a80a2e9a1e187639df1bb0846d2a3bedd56604be906b66dd60714382cc
Sep 10 19:12:01.210388 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.210291    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx" event={"ID":"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea","Type":"ContainerStarted","Data":"907f57a80a2e9a1e187639df1bb0846d2a3bedd56604be906b66dd60714382cc"}
Sep 10 19:12:01.212419 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.212392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5" event={"ID":"8c542347-3526-4891-acd3-a052ded42597","Type":"ContainerStarted","Data":"a872114604bd936d51e1652636574c87f70eba6682d5fd19cbee5f541d3aad40"}
Sep 10 19:12:01.212518 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.212442    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5" event={"ID":"8c542347-3526-4891-acd3-a052ded42597","Type":"ContainerStarted","Data":"85b24f5cf193fc11ddad6bc864fd39d35b54b9d32c4fa0e68be46e78fa01e72b"}
Sep 10 19:12:01.212518 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.212473    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5" event={"ID":"8c542347-3526-4891-acd3-a052ded42597","Type":"ContainerStarted","Data":"778855189ed0ec413984b944ed1428566b6b1f177ae487ba193081449fc50908"}
Sep 10 19:12:01.213491 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.213470    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b7tbz" event={"ID":"a9f1b138-b565-4acd-a676-568aa20a8321","Type":"ContainerStarted","Data":"c3d7b0f3e4c4d31fff7d0c34eeb256153d9cb64356b4661b7a09ab08182b86aa"}
Sep 10 19:12:01.325171 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.325127    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 19:12:01.329655 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.329566    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.333773 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.333750    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 19:12:01.334062 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.334039    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 19:12:01.334190 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.334039    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 19:12:01.334405 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.334385    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vjllq\""
Sep 10 19:12:01.334482 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.334447    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 19:12:01.334581 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.334543    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 19:12:01.334636 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.334588    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 19:12:01.334636 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.334624    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 19:12:01.334899 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.334879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 19:12:01.334988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.334943    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 19:12:01.361257 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.361164    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 19:12:01.462807 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.462721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.462807 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.462795    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.462830    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.462855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.462907    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.462940    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.462996    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463231 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.463031    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-out\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463231 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.463059    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxz4n\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-kube-api-access-sxz4n\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463231 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.463091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463231 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.463116    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463231 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.463218    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-web-config\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.463394 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.463277    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.563693 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563650    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.563878 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.563878 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563813    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.563878 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563835    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.563878 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563902    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563931    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563946    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-out\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563964    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxz4n\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-kube-api-access-sxz4n\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.563984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.564004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.564019    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-web-config\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.564048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564475 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.564110    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.564475 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:12:01.564178    2573 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 10 19:12:01.564475 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:12:01.564238    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-main-tls podName:ca06c060-a6e2-498e-bbe8-dd81f7f3194b nodeName:}" failed. No retries permitted until 2026-09-10 19:12:02.064219895 +0000 UTC m=+57.724017081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b") : secret "alertmanager-main-tls" not found
Sep 10 19:12:01.564711 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.564689    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.566026 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.565872    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.567572 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.567522    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-volume\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.568418 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.568136    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.568602 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.568563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.568684 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.568607    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.568744 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.568695    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-web-config\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.568922 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.568894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.569647 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.569627    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.569737 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.569718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-out\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:01.587504 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:01.587478    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxz4n\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-kube-api-access-sxz4n\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:02.070008 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.069790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:02.073366 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.073326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:02.240646 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.240612    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:12:02.347804 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.347725    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"]
Sep 10 19:12:02.351774 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.351742    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.357687 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.357665    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 19:12:02.358917 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.358892    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 19:12:02.359471 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.358988    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 19:12:02.359471 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.359216    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4vd483nk9e5f8\""
Sep 10 19:12:02.359471 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.359319    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 19:12:02.359648 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.359474    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 19:12:02.359648 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.359530    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rhvk8\""
Sep 10 19:12:02.371314 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.371285    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"]
Sep 10 19:12:02.473719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.473673    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.473910 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.473757    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.473910 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.473806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a163d3dd-5faa-488a-bb2a-af839633f2bb-metrics-client-ca\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.473910 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.473832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzc7\" (UniqueName: \"kubernetes.io/projected/a163d3dd-5faa-488a-bb2a-af839633f2bb-kube-api-access-wlzc7\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.473910 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.473884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.474070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.473925    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-tls\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.474070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.473971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.474070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.474012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-grpc-tls\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.575043 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.574994    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.575221 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.575062    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-tls\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.575221 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.575110    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.575221 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.575170    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-grpc-tls\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.575221 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.575214    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.575423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.575266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.575423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.575290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a163d3dd-5faa-488a-bb2a-af839633f2bb-metrics-client-ca\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.575423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.575314    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzc7\" (UniqueName: \"kubernetes.io/projected/a163d3dd-5faa-488a-bb2a-af839633f2bb-kube-api-access-wlzc7\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.577340 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.577307    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a163d3dd-5faa-488a-bb2a-af839633f2bb-metrics-client-ca\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.579303 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.579267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.581180 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.581129    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-grpc-tls\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.581272 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.581181    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.582242 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.581550    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.582242 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.581866    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.582242 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.582203    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a163d3dd-5faa-488a-bb2a-af839633f2bb-secret-thanos-querier-tls\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.585317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.585293    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzc7\" (UniqueName: \"kubernetes.io/projected/a163d3dd-5faa-488a-bb2a-af839633f2bb-kube-api-access-wlzc7\") pod \"thanos-querier-5b5bf859fb-m9ws4\" (UID: \"a163d3dd-5faa-488a-bb2a-af839633f2bb\") " pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:02.664185 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:02.663859    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:03.225760 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.225663    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5" event={"ID":"8c542347-3526-4891-acd3-a052ded42597","Type":"ContainerStarted","Data":"3b366fe841256c104f70ec954a78f17be81f0e66bf408a26ef7838ea0741cb47"}
Sep 10 19:12:03.228568 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.228499    2573 generic.go:356] "Generic (PLEG): container finished" podID="a9f1b138-b565-4acd-a676-568aa20a8321" containerID="6e8aeff2baaf9f217afbef3ed5c9f3c67eca433c6d5357881f108ca226f1b50f" exitCode=0
Sep 10 19:12:03.228703 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.228641    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b7tbz" event={"ID":"a9f1b138-b565-4acd-a676-568aa20a8321","Type":"ContainerDied","Data":"6e8aeff2baaf9f217afbef3ed5c9f3c67eca433c6d5357881f108ca226f1b50f"}
Sep 10 19:12:03.234535 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.234406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx" event={"ID":"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea","Type":"ContainerStarted","Data":"9b9f8e5b948f2a0f7471e5489ddf7ba6fbff57fe9d8dbc3987e9e9e2b5aae007"}
Sep 10 19:12:03.234535 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.234499    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx" event={"ID":"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea","Type":"ContainerStarted","Data":"1b3c74d85871183ded2ae639d0a416314c3d2817e2bde831c68ec7a5b7af0b75"}
Sep 10 19:12:03.253104 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.253046    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2qnp5" podStartSLOduration=1.275178302 podStartE2EDuration="3.253022552s" totalImagesPullingTime="1.97784425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:03.251971262 +0000 UTC m=+58.911768455" watchObservedRunningTime="2026-09-10 19:12:03.253022552 +0000 UTC m=+58.912819748"
Sep 10 19:12:03.288493 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.288350    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"]
Sep 10 19:12:03.289452 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.289426    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 19:12:03.300029 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:03.299947    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda163d3dd_5faa_488a_bb2a_af839633f2bb.slice/crio-0eb314cbcfb9617021d36c9730e3f62f299350a631a9a0ad42cb7fc065875d22 WatchSource:0}: Error finding container 0eb314cbcfb9617021d36c9730e3f62f299350a631a9a0ad42cb7fc065875d22: Status 404 returned error can't find the container with id 0eb314cbcfb9617021d36c9730e3f62f299350a631a9a0ad42cb7fc065875d22
Sep 10 19:12:03.300482 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:03.300452    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca06c060_a6e2_498e_bbe8_dd81f7f3194b.slice/crio-4a4dceae010dcbf00693e992a0a83c2a5e1fda8c10a995eae83e9de9e9e49984 WatchSource:0}: Error finding container 4a4dceae010dcbf00693e992a0a83c2a5e1fda8c10a995eae83e9de9e9e49984: Status 404 returned error can't find the container with id 4a4dceae010dcbf00693e992a0a83c2a5e1fda8c10a995eae83e9de9e9e49984
Sep 10 19:12:03.638656 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.636686    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf9677f97-4wng5"]
Sep 10 19:12:03.656234 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.654864    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf9677f97-4wng5"]
Sep 10 19:12:03.656234 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.654981    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.658035 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.658008    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 19:12:03.658201 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.658007    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fnqjb\""
Sep 10 19:12:03.658201 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.658099    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 19:12:03.658302 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.658248    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 19:12:03.658822 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.658797    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 19:12:03.658934 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.658836    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 19:12:03.786903 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.786868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-oauth-serving-cert\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.787108 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.787002    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-serving-cert\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.787108 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.787036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-oauth-config\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.787108 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.787086    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-service-ca\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.787269 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.787178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2gnc\" (UniqueName: \"kubernetes.io/projected/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-kube-api-access-j2gnc\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.787269 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.787228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-config\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.888777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.888691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-service-ca\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.888777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.888755    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2gnc\" (UniqueName: \"kubernetes.io/projected/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-kube-api-access-j2gnc\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.889005 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.888793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-config\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.889005 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.888864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-oauth-serving-cert\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.889005 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.888927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-serving-cert\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.889005 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.888951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-oauth-config\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.890194 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.889456    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-service-ca\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.890194 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.889961    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-config\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.890518 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.890476    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-oauth-serving-cert\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.892168 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.892125    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-oauth-config\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.892488 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.892470    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-serving-cert\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.898348 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.898324    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2gnc\" (UniqueName: \"kubernetes.io/projected/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-kube-api-access-j2gnc\") pod \"console-7bf9677f97-4wng5\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") " pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:03.968919 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:03.968872    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:04.112612 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.112580    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxrqg"
Sep 10 19:12:04.245059 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.244136    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b7tbz" event={"ID":"a9f1b138-b565-4acd-a676-568aa20a8321","Type":"ContainerStarted","Data":"a26dbc797860423636419d988a59a0de9d4c657cf80d771849de20c5dda8ca5e"}
Sep 10 19:12:04.245059 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.244229    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b7tbz" event={"ID":"a9f1b138-b565-4acd-a676-568aa20a8321","Type":"ContainerStarted","Data":"ea786e37f42e34bc79534b16ab94826f9720815e113048e18719a8b7b9b616e7"}
Sep 10 19:12:04.247174 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.247121    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4" event={"ID":"a163d3dd-5faa-488a-bb2a-af839633f2bb","Type":"ContainerStarted","Data":"0eb314cbcfb9617021d36c9730e3f62f299350a631a9a0ad42cb7fc065875d22"}
Sep 10 19:12:04.251575 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.250891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx" event={"ID":"f5330f3a-ae25-4efa-92b6-fb2e6f0054ea","Type":"ContainerStarted","Data":"1260b5b9e4def8847aea59371a594f6d2af1bf5d4a495b4bfef17164562b1ab7"}
Sep 10 19:12:04.254054 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.253977    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerStarted","Data":"4a4dceae010dcbf00693e992a0a83c2a5e1fda8c10a995eae83e9de9e9e49984"}
Sep 10 19:12:04.276464 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.276400    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b7tbz" podStartSLOduration=1.7633409 podStartE2EDuration="4.276380554s" totalImagesPullingTime="2.424203675s" totalInitContainerRuntime="88.835979ms" isStatefulPod=true podCreationTimestamp="2026-09-10 19:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:04.275416433 +0000 UTC m=+59.935213628" watchObservedRunningTime="2026-09-10 19:12:04.276380554 +0000 UTC m=+59.936177749"
Sep 10 19:12:04.297321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.297248    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf9677f97-4wng5"]
Sep 10 19:12:04.301964 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:04.301937    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d6aa9b_6612_4b75_aa89_510966e3a4ae.slice/crio-d0ad3df2890d0ad2c4ab85d7dd07532434da49ebf4f0b3d89af067dfd001735b WatchSource:0}: Error finding container d0ad3df2890d0ad2c4ab85d7dd07532434da49ebf4f0b3d89af067dfd001735b: Status 404 returned error can't find the container with id d0ad3df2890d0ad2c4ab85d7dd07532434da49ebf4f0b3d89af067dfd001735b
Sep 10 19:12:04.305570 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.305504    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-fh2tx" podStartSLOduration=2.181168371 podStartE2EDuration="4.305483603s" totalImagesPullingTime="2.124315232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:04.304959291 +0000 UTC m=+59.964756495" watchObservedRunningTime="2026-09-10 19:12:04.305483603 +0000 UTC m=+59.965280790"
Sep 10 19:12:04.562670 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.562635    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-59597c4497-bl8ch"]
Sep 10 19:12:04.573066 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.573029    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.576006 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.575938    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59597c4497-bl8ch"]
Sep 10 19:12:04.576301 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.576047    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 19:12:04.576377 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.576323    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 19:12:04.577441 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.577418    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vdthd\""
Sep 10 19:12:04.577670 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.577650    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 19:12:04.578489 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.578467    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-51c9itr6459d0\""
Sep 10 19:12:04.578819 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.578800    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 19:12:04.698339 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.698300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f7b63c18-d466-49fe-90c3-aa770bc5695a-secret-metrics-server-client-certs\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.698521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.698349    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f7b63c18-d466-49fe-90c3-aa770bc5695a-secret-metrics-server-tls\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.698521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.698413    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b63c18-d466-49fe-90c3-aa770bc5695a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.698521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.698489    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f7b63c18-d466-49fe-90c3-aa770bc5695a-metrics-server-audit-profiles\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.698684 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.698585    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f7b63c18-d466-49fe-90c3-aa770bc5695a-client-ca-bundle\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.698684 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.698642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f7b63c18-d466-49fe-90c3-aa770bc5695a-audit-log\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.698684 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.698667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzqc\" (UniqueName: \"kubernetes.io/projected/f7b63c18-d466-49fe-90c3-aa770bc5695a-kube-api-access-dhzqc\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.800062 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.800013    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f7b63c18-d466-49fe-90c3-aa770bc5695a-client-ca-bundle\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.800230 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.800076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f7b63c18-d466-49fe-90c3-aa770bc5695a-audit-log\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.800230 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.800112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzqc\" (UniqueName: \"kubernetes.io/projected/f7b63c18-d466-49fe-90c3-aa770bc5695a-kube-api-access-dhzqc\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.800230 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.800196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f7b63c18-d466-49fe-90c3-aa770bc5695a-secret-metrics-server-client-certs\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.800407 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.800229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f7b63c18-d466-49fe-90c3-aa770bc5695a-secret-metrics-server-tls\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.800407 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.800289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b63c18-d466-49fe-90c3-aa770bc5695a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.800407 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.800328    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f7b63c18-d466-49fe-90c3-aa770bc5695a-metrics-server-audit-profiles\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.801050 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.801028    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f7b63c18-d466-49fe-90c3-aa770bc5695a-audit-log\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.803289 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.803262    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 10 19:12:04.803396 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.803293    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 10 19:12:04.803643 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.803620    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-51c9itr6459d0\""
Sep 10 19:12:04.803741 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.803704    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 10 19:12:04.803802 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.803766    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 19:12:04.812343 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.812265    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7b63c18-d466-49fe-90c3-aa770bc5695a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.812704 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.812648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f7b63c18-d466-49fe-90c3-aa770bc5695a-metrics-server-audit-profiles\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.816314 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.816289    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f7b63c18-d466-49fe-90c3-aa770bc5695a-client-ca-bundle\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.820642 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.820617    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzqc\" (UniqueName: \"kubernetes.io/projected/f7b63c18-d466-49fe-90c3-aa770bc5695a-kube-api-access-dhzqc\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.822855 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.822806    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f7b63c18-d466-49fe-90c3-aa770bc5695a-secret-metrics-server-client-certs\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.823478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.823457    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f7b63c18-d466-49fe-90c3-aa770bc5695a-secret-metrics-server-tls\") pod \"metrics-server-59597c4497-bl8ch\" (UID: \"f7b63c18-d466-49fe-90c3-aa770bc5695a\") " pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.896299 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.896264    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vdthd\""
Sep 10 19:12:04.903231 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.903205    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:04.982133 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.982099    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"]
Sep 10 19:12:04.998294 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.998263    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"]
Sep 10 19:12:04.998468 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:04.998403    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:05.001576 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.001313    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 19:12:05.001826 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.001792    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bksk2\""
Sep 10 19:12:05.102763 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.102723    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fr5r\" (UniqueName: \"kubernetes.io/projected/4f7af8f8-1261-4293-98b4-597e20e24f1b-kube-api-access-8fr5r\") pod \"monitoring-plugin-85bbbbbf4-z8rjz\" (UID: \"4f7af8f8-1261-4293-98b4-597e20e24f1b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:05.102964 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.102943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f7af8f8-1261-4293-98b4-597e20e24f1b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-z8rjz\" (UID: \"4f7af8f8-1261-4293-98b4-597e20e24f1b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:05.204233 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.204186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f7af8f8-1261-4293-98b4-597e20e24f1b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-z8rjz\" (UID: \"4f7af8f8-1261-4293-98b4-597e20e24f1b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:05.204403 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.204241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fr5r\" (UniqueName: \"kubernetes.io/projected/4f7af8f8-1261-4293-98b4-597e20e24f1b-kube-api-access-8fr5r\") pod \"monitoring-plugin-85bbbbbf4-z8rjz\" (UID: \"4f7af8f8-1261-4293-98b4-597e20e24f1b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:05.204589 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:12:05.204562    2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 10 19:12:05.204706 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:12:05.204637    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7af8f8-1261-4293-98b4-597e20e24f1b-monitoring-plugin-cert podName:4f7af8f8-1261-4293-98b4-597e20e24f1b nodeName:}" failed. No retries permitted until 2026-09-10 19:12:05.704621244 +0000 UTC m=+61.364418416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/4f7af8f8-1261-4293-98b4-597e20e24f1b-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-z8rjz" (UID: "4f7af8f8-1261-4293-98b4-597e20e24f1b") : secret "monitoring-plugin-cert" not found
Sep 10 19:12:05.214876 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.214845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fr5r\" (UniqueName: \"kubernetes.io/projected/4f7af8f8-1261-4293-98b4-597e20e24f1b-kube-api-access-8fr5r\") pod \"monitoring-plugin-85bbbbbf4-z8rjz\" (UID: \"4f7af8f8-1261-4293-98b4-597e20e24f1b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:05.261715 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.261674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf9677f97-4wng5" event={"ID":"b7d6aa9b-6612-4b75-aa89-510966e3a4ae","Type":"ContainerStarted","Data":"d0ad3df2890d0ad2c4ab85d7dd07532434da49ebf4f0b3d89af067dfd001735b"}
Sep 10 19:12:05.264957 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.264923    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerStarted","Data":"1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec"}
Sep 10 19:12:05.285714 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.285676    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59597c4497-bl8ch"]
Sep 10 19:12:05.442994 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.442905    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d56dd9545-wr64z"]
Sep 10 19:12:05.447650 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.447623    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.450823 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.450802    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 19:12:05.451039 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.450809    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 19:12:05.452710 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.452267    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 19:12:05.452710 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.452301    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 19:12:05.452710 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.452475    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-q9txz\""
Sep 10 19:12:05.453462 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.453281    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 19:12:05.457924 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.457773    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 19:12:05.462409 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.462335    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d56dd9545-wr64z"]
Sep 10 19:12:05.608742 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.608705    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-secret-telemeter-client\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.608928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.608861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070da58d-1daa-4ed3-90af-f04b701d9a2c-serving-certs-ca-bundle\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.608928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.608888    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.608928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.608910    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070da58d-1daa-4ed3-90af-f04b701d9a2c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.609092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.608935    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-telemeter-client-tls\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.609092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.608950    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5kq\" (UniqueName: \"kubernetes.io/projected/070da58d-1daa-4ed3-90af-f04b701d9a2c-kube-api-access-kt5kq\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.609092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.608976    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-federate-client-tls\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.609092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.609001    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070da58d-1daa-4ed3-90af-f04b701d9a2c-metrics-client-ca\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.710327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.710228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f7af8f8-1261-4293-98b4-597e20e24f1b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-z8rjz\" (UID: \"4f7af8f8-1261-4293-98b4-597e20e24f1b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:05.710327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.710291    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070da58d-1daa-4ed3-90af-f04b701d9a2c-serving-certs-ca-bundle\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.710327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.710321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.710644 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.710618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070da58d-1daa-4ed3-90af-f04b701d9a2c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.710695 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.710674    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-telemeter-client-tls\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.710757 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.710704    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5kq\" (UniqueName: \"kubernetes.io/projected/070da58d-1daa-4ed3-90af-f04b701d9a2c-kube-api-access-kt5kq\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.710757 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.710734    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-federate-client-tls\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.710854 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.710757    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070da58d-1daa-4ed3-90af-f04b701d9a2c-metrics-client-ca\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.711693 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.710904    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-secret-telemeter-client\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.711693 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.711083    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070da58d-1daa-4ed3-90af-f04b701d9a2c-serving-certs-ca-bundle\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.711693 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.711440    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/070da58d-1daa-4ed3-90af-f04b701d9a2c-metrics-client-ca\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.711693 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.711570    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/070da58d-1daa-4ed3-90af-f04b701d9a2c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.714086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.714056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.715113 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.715040    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-secret-telemeter-client\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.717959 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.716359    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-telemeter-client-tls\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.717959 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.716863    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/070da58d-1daa-4ed3-90af-f04b701d9a2c-federate-client-tls\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.718450 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.718421    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4f7af8f8-1261-4293-98b4-597e20e24f1b-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-z8rjz\" (UID: \"4f7af8f8-1261-4293-98b4-597e20e24f1b\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:05.728664 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.728604    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5kq\" (UniqueName: \"kubernetes.io/projected/070da58d-1daa-4ed3-90af-f04b701d9a2c-kube-api-access-kt5kq\") pod \"telemeter-client-5d56dd9545-wr64z\" (UID: \"070da58d-1daa-4ed3-90af-f04b701d9a2c\") " pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.762917 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.762875    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z"
Sep 10 19:12:05.918130 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:05.918085    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:05.980597 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:05.980509    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b63c18_d466_49fe_90c3_aa770bc5695a.slice/crio-9eca765b1b3101ce28894b70aacfb8dff1ca8fbaa6daa551f52ae395cb8f7a99 WatchSource:0}: Error finding container 9eca765b1b3101ce28894b70aacfb8dff1ca8fbaa6daa551f52ae395cb8f7a99: Status 404 returned error can't find the container with id 9eca765b1b3101ce28894b70aacfb8dff1ca8fbaa6daa551f52ae395cb8f7a99
Sep 10 19:12:06.194904 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.194874    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b8fgm"
Sep 10 19:12:06.281953 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.281518    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4" event={"ID":"a163d3dd-5faa-488a-bb2a-af839633f2bb","Type":"ContainerStarted","Data":"eafacc0cbdb0252ae0a7a57ade81235130614ec50b61d3779c637de8c18a0d02"}
Sep 10 19:12:06.281953 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.281600    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4" event={"ID":"a163d3dd-5faa-488a-bb2a-af839633f2bb","Type":"ContainerStarted","Data":"1e0243ae76bf116efc2bbe90a6af003734cc3d85d3ce35bfb46b6e0f363f2411"}
Sep 10 19:12:06.283771 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.283675    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"]
Sep 10 19:12:06.285843 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.285809    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59597c4497-bl8ch" event={"ID":"f7b63c18-d466-49fe-90c3-aa770bc5695a","Type":"ContainerStarted","Data":"9eca765b1b3101ce28894b70aacfb8dff1ca8fbaa6daa551f52ae395cb8f7a99"}
Sep 10 19:12:06.290107 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.290081    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d56dd9545-wr64z"]
Sep 10 19:12:06.292384 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.292361    2573 generic.go:356] "Generic (PLEG): container finished" podID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerID="1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec" exitCode=0
Sep 10 19:12:06.292624 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.292440    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerDied","Data":"1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec"}
Sep 10 19:12:06.308405 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:06.308374    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070da58d_1daa_4ed3_90af_f04b701d9a2c.slice/crio-198a669e865220f9f7bc3ffe30c6bdd3d5be23f7034a0b7bc28e98550d5a2946 WatchSource:0}: Error finding container 198a669e865220f9f7bc3ffe30c6bdd3d5be23f7034a0b7bc28e98550d5a2946: Status 404 returned error can't find the container with id 198a669e865220f9f7bc3ffe30c6bdd3d5be23f7034a0b7bc28e98550d5a2946
Sep 10 19:12:06.627181 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.626313    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 19:12:06.631080 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.631034    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.634013 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.633985    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 19:12:06.634228 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.634041    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 19:12:06.634414 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.634394    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 19:12:06.635374 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.634742    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 19:12:06.635374 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.634792    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 19:12:06.635534 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.635377    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 19:12:06.635773 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.635607    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fog6mbhi4hoic\""
Sep 10 19:12:06.641020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.637109    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 19:12:06.641020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.637392    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 19:12:06.641020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.637508    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 19:12:06.641020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.637540    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5bzmk\""
Sep 10 19:12:06.641020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.637670    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 19:12:06.641020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.637735    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 19:12:06.641020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.637857    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 19:12:06.641020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.640653    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 19:12:06.655131 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.655097    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 19:12:06.657294 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.657266    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 19:12:06.826252 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826210    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826260    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826292    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826316    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826336    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826381    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjk2r\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-kube-api-access-vjk2r\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826443    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826758 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826758 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826643    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826758 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826708    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826899 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826899 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826899 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826853    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.826899 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.827078 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.827078 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.826967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.827078 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.827068    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-config\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.827249 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.827116    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.928675 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.928575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.928844 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.928703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.928844 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.928729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.928844 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.928747    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.928844 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.928763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.929041 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.928778    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.929041 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.928978    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.929041 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjk2r\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-kube-api-access-vjk2r\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929670    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929851    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929900    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929928    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.929996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.930336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.930029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-config\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.935650 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.932357    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.935650 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.932774    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.935650 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.935267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.935650 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.935438    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.936389 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.936173    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.937403 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.936794    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.937403 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.937107    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.937552 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.937494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.939614 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.939552    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.941113 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.941086    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.944640 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.944613    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.946895 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.946431    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.947481 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.947423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.948051 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.948009    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.948137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.948062    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.948495 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.948456    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.948918 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.948896    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.950761 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.950720    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjk2r\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-kube-api-access-vjk2r\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.950934 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.950915    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-config\") pod \"prometheus-k8s-0\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:06.966591 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:06.965780    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:07.281031 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:07.280922    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 19:12:07.285746 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:07.285706    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39748edc_cb63_41e8_b9c0_2c13417157cb.slice/crio-2de07de16cbfe86fd8aadbb0d2513c69ba96004c15a76aca37c75b7cbc4b678e WatchSource:0}: Error finding container 2de07de16cbfe86fd8aadbb0d2513c69ba96004c15a76aca37c75b7cbc4b678e: Status 404 returned error can't find the container with id 2de07de16cbfe86fd8aadbb0d2513c69ba96004c15a76aca37c75b7cbc4b678e
Sep 10 19:12:07.310458 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:07.310358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz" event={"ID":"4f7af8f8-1261-4293-98b4-597e20e24f1b","Type":"ContainerStarted","Data":"ce2ae84670d5cebeedb794f9cedd37517fed200b552ec7614cde37cf20d82a68"}
Sep 10 19:12:07.318515 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:07.318419    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z" event={"ID":"070da58d-1daa-4ed3-90af-f04b701d9a2c","Type":"ContainerStarted","Data":"198a669e865220f9f7bc3ffe30c6bdd3d5be23f7034a0b7bc28e98550d5a2946"}
Sep 10 19:12:07.324579 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:07.324538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerStarted","Data":"2de07de16cbfe86fd8aadbb0d2513c69ba96004c15a76aca37c75b7cbc4b678e"}
Sep 10 19:12:07.330066 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:07.330030    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4" event={"ID":"a163d3dd-5faa-488a-bb2a-af839633f2bb","Type":"ContainerStarted","Data":"0370b1d894dc6ea813b76ac680416987167739a0cea7811565074e887db943b6"}
Sep 10 19:12:08.338206 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:08.338098    2573 generic.go:356] "Generic (PLEG): container finished" podID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerID="bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf" exitCode=0
Sep 10 19:12:08.339699 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:08.339296    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerDied","Data":"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"}
Sep 10 19:12:09.557124 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.557084    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:12:09.560075 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.560037    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 19:12:09.570872 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.570842    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46be1c16-0b44-44ae-8675-69b7b8442231-metrics-certs\") pod \"network-metrics-daemon-5v6bv\" (UID: \"46be1c16-0b44-44ae-8675-69b7b8442231\") " pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:12:09.658278 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.658244    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:12:09.661668 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.661633    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 19:12:09.671635 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.671606    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfc622a4-f58d-42ec-97f6-5271a1a9d3bc-original-pull-secret\") pod \"global-pull-secret-syncer-htz6q\" (UID: \"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc\") " pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:12:09.758826 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.758790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:12:09.761685 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.761656    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 19:12:09.772106 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.772082    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 19:12:09.783813 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.783773    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hfl\" (UniqueName: \"kubernetes.io/projected/c15a2707-8b56-4c41-8d37-98cd53e9d4e6-kube-api-access-67hfl\") pod \"network-check-target-qlmln\" (UID: \"c15a2707-8b56-4c41-8d37-98cd53e9d4e6\") " pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:12:09.837036 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.836948    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-htz6q"
Sep 10 19:12:09.846040 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.846019    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zj5gq\""
Sep 10 19:12:09.849279 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.849260    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8gdr\""
Sep 10 19:12:09.853225 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.853204    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:12:09.858076 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:09.858051    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5v6bv"
Sep 10 19:12:19.968537 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:19.966178    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-htz6q"]
Sep 10 19:12:19.968537 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:19.968246    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qlmln"]
Sep 10 19:12:19.973140 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:19.972192    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5v6bv"]
Sep 10 19:12:19.975324 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:19.975284    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15a2707_8b56_4c41_8d37_98cd53e9d4e6.slice/crio-80842283293a221daa612ea814d5121062a95d9c1ad54280f77fb2dd64864af5 WatchSource:0}: Error finding container 80842283293a221daa612ea814d5121062a95d9c1ad54280f77fb2dd64864af5: Status 404 returned error can't find the container with id 80842283293a221daa612ea814d5121062a95d9c1ad54280f77fb2dd64864af5
Sep 10 19:12:19.975885 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:12:19.975658    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46be1c16_0b44_44ae_8675_69b7b8442231.slice/crio-1d7f6ee5f1d45f6d8fb838fe9d9565c7042595423541ef3a8551211c32de2cd0 WatchSource:0}: Error finding container 1d7f6ee5f1d45f6d8fb838fe9d9565c7042595423541ef3a8551211c32de2cd0: Status 404 returned error can't find the container with id 1d7f6ee5f1d45f6d8fb838fe9d9565c7042595423541ef3a8551211c32de2cd0
Sep 10 19:12:20.396226 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.395512    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf9677f97-4wng5" event={"ID":"b7d6aa9b-6612-4b75-aa89-510966e3a4ae","Type":"ContainerStarted","Data":"d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a"}
Sep 10 19:12:20.406234 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.404017    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59597c4497-bl8ch" event={"ID":"f7b63c18-d466-49fe-90c3-aa770bc5695a","Type":"ContainerStarted","Data":"2f1430d72843906f10d1d704b8b2af39a680f316892a3803b69c5d703ad26489"}
Sep 10 19:12:20.409217 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.406677    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-6vdrv" event={"ID":"41c2f980-3cbf-4fe6-8daf-d90a57fec390","Type":"ContainerStarted","Data":"29cf39fc7341c9e5aca9c266928482764e7590882de07d41d1f7927250f928f8"}
Sep 10 19:12:20.409217 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.408230    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-6vdrv"
Sep 10 19:12:20.409217 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.408902    2573 patch_prober.go:28] interesting pod/downloads-85d98c998f-6vdrv container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.9:8080/\": dial tcp 10.134.0.9:8080: connect: connection refused" start-of-body=
Sep 10 19:12:20.409217 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.408951    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-6vdrv" podUID="41c2f980-3cbf-4fe6-8daf-d90a57fec390" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.9:8080/\": dial tcp 10.134.0.9:8080: connect: connection refused"
Sep 10 19:12:20.414111 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.414032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz" event={"ID":"4f7af8f8-1261-4293-98b4-597e20e24f1b","Type":"ContainerStarted","Data":"6f4ec443839cee672c1fedadc1a9447779550f13c8633e795a9d8763498c041d"}
Sep 10 19:12:20.415225 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.415167    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:20.418240 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.418014    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf9677f97-4wng5" podStartSLOduration=2.065151418 podStartE2EDuration="17.417996498s" totalImagesPullingTime="15.35284508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:20.416776897 +0000 UTC m=+76.076574091" watchObservedRunningTime="2026-09-10 19:12:20.417996498 +0000 UTC m=+76.077793708"
Sep 10 19:12:20.425086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.420813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z" event={"ID":"070da58d-1daa-4ed3-90af-f04b701d9a2c","Type":"ContainerStarted","Data":"546ad6899667799bfdb9d6e5b2bbde235d7e478b04760994cbba514d9caea6fa"}
Sep 10 19:12:20.425086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.420868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z" event={"ID":"070da58d-1daa-4ed3-90af-f04b701d9a2c","Type":"ContainerStarted","Data":"52d6fde52105af99e0da8bee0118b32d700f3d835019b647179579b8ba0ed957"}
Sep 10 19:12:20.425086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.420896    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z" event={"ID":"070da58d-1daa-4ed3-90af-f04b701d9a2c","Type":"ContainerStarted","Data":"8c6d6d7517e0c0991aced8db9c96e11da44da09623dfadc4a49731302021a818"}
Sep 10 19:12:20.425086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.425041    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz"
Sep 10 19:12:20.434706 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.434675    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4" event={"ID":"a163d3dd-5faa-488a-bb2a-af839633f2bb","Type":"ContainerStarted","Data":"e371becb58be31b992bebe3bcf8184c87fd5cb198bda586709eff17b04e2603b"}
Sep 10 19:12:20.434828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.434760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4" event={"ID":"a163d3dd-5faa-488a-bb2a-af839633f2bb","Type":"ContainerStarted","Data":"09f72d05db20af4b970c6122fa57f9e2335905afd389ce5d7f28207dca80aa3b"}
Sep 10 19:12:20.434828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.434804    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4" event={"ID":"a163d3dd-5faa-488a-bb2a-af839633f2bb","Type":"ContainerStarted","Data":"0c68570c181c0e231ac6d9837ed6a7c15bab7ad4542f7d7a28edc891cf7cadc4"}
Sep 10 19:12:20.435697 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.435667    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:20.439987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.439026    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-59597c4497-bl8ch" podStartSLOduration=2.76401858 podStartE2EDuration="16.439007483s" totalImagesPullingTime="13.674988903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:20.437702096 +0000 UTC m=+76.097499290" watchObservedRunningTime="2026-09-10 19:12:20.439007483 +0000 UTC m=+76.098804675"
Sep 10 19:12:20.443260 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.443189    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-htz6q" event={"ID":"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc","Type":"ContainerStarted","Data":"3f847dedd0182c4175b17bf2553793e3c1cf859ddcdd5c924d58de632e5f5458"}
Sep 10 19:12:20.446575 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.446528    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4"
Sep 10 19:12:20.447330 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.447285    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5v6bv" event={"ID":"46be1c16-0b44-44ae-8675-69b7b8442231","Type":"ContainerStarted","Data":"1d7f6ee5f1d45f6d8fb838fe9d9565c7042595423541ef3a8551211c32de2cd0"}
Sep 10 19:12:20.450651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.450407    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerStarted","Data":"02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162"}
Sep 10 19:12:20.450651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.450474    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerStarted","Data":"990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac"}
Sep 10 19:12:20.450651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.450514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerStarted","Data":"657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb"}
Sep 10 19:12:20.450651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.450555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerStarted","Data":"e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42"}
Sep 10 19:12:20.450651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.450593    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerStarted","Data":"03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1"}
Sep 10 19:12:20.455048 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.455020    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerStarted","Data":"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"}
Sep 10 19:12:20.455163 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.455093    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerStarted","Data":"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"}
Sep 10 19:12:20.455163 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.455134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerStarted","Data":"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"}
Sep 10 19:12:20.455245 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.455210    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerStarted","Data":"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"}
Sep 10 19:12:20.458376 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.458328    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-6vdrv" podStartSLOduration=2.527419822 podStartE2EDuration="26.458307253s" totalImagesPullingTime="23.930887431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:20.457799142 +0000 UTC m=+76.117596328" watchObservedRunningTime="2026-09-10 19:12:20.458307253 +0000 UTC m=+76.118104447"
Sep 10 19:12:20.458524 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.458497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qlmln" event={"ID":"c15a2707-8b56-4c41-8d37-98cd53e9d4e6","Type":"ContainerStarted","Data":"80842283293a221daa612ea814d5121062a95d9c1ad54280f77fb2dd64864af5"}
Sep 10 19:12:20.506283 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.506212    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-z8rjz" podStartSLOduration=3.159079553 podStartE2EDuration="16.506190507s" totalImagesPullingTime="13.347110954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:20.480704172 +0000 UTC m=+76.140501367" watchObservedRunningTime="2026-09-10 19:12:20.506190507 +0000 UTC m=+76.165987707"
Sep 10 19:12:20.506941 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.506903    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d56dd9545-wr64z" podStartSLOduration=2.143555613 podStartE2EDuration="15.506871309s" totalImagesPullingTime="13.363315696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:20.502983537 +0000 UTC m=+76.162780730" watchObservedRunningTime="2026-09-10 19:12:20.506871309 +0000 UTC m=+76.166668509"
Sep 10 19:12:20.553209 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.549506    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b5bf859fb-m9ws4" podStartSLOduration=3.596143767 podStartE2EDuration="18.549479109s" totalImagesPullingTime="14.953335342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:20.530615168 +0000 UTC m=+76.190412374" watchObservedRunningTime="2026-09-10 19:12:20.549479109 +0000 UTC m=+76.209276303"
Sep 10 19:12:20.607190 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:20.607111    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf9677f97-4wng5"]
Sep 10 19:12:21.479706 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:21.479597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerStarted","Data":"ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035"}
Sep 10 19:12:21.488456 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:21.487943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerStarted","Data":"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"}
Sep 10 19:12:21.488456 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:21.488017    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerStarted","Data":"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"}
Sep 10 19:12:21.493936 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:21.493874    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-6vdrv"
Sep 10 19:12:21.526439 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:21.526221    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.441130389 podStartE2EDuration="20.526196496s" totalImagesPullingTime="15.067083766s" totalInitContainerRuntime="17.982341ms" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:21.524568468 +0000 UTC m=+77.184365675" watchObservedRunningTime="2026-09-10 19:12:21.526196496 +0000 UTC m=+77.185993692"
Sep 10 19:12:21.582882 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:21.581357    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.110398828 podStartE2EDuration="15.581328345s" totalImagesPullingTime="11.439486833s" totalInitContainerRuntime="31.442684ms" isStatefulPod=false podCreationTimestamp="2026-09-10 19:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:21.578537926 +0000 UTC m=+77.238335324" watchObservedRunningTime="2026-09-10 19:12:21.581328345 +0000 UTC m=+77.241125539"
Sep 10 19:12:21.966935 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:21.966777    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:12:23.505682 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:23.505641    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5v6bv" event={"ID":"46be1c16-0b44-44ae-8675-69b7b8442231","Type":"ContainerStarted","Data":"a8c9350624075010591ea3b7ebcdac40e67745eeca8634f8244022c41a72da72"}
Sep 10 19:12:23.506130 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:23.505722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5v6bv" event={"ID":"46be1c16-0b44-44ae-8675-69b7b8442231","Type":"ContainerStarted","Data":"3cda6c00ea0a3a367ad38f3fd62b3ad7e48687179ae22b54d493e1b178eeb8d3"}
Sep 10 19:12:23.527476 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:23.527329    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5v6bv" podStartSLOduration=76.927808553 podStartE2EDuration="1m19.527302154s" totalImagesPullingTime="2.599493601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:23.523096658 +0000 UTC m=+79.182893863" watchObservedRunningTime="2026-09-10 19:12:23.527302154 +0000 UTC m=+79.187099348"
Sep 10 19:12:23.969886 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:23.969803    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:24.904301 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:24.904253    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:24.904834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:24.904359    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:27.529394 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:27.529344    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qlmln" event={"ID":"c15a2707-8b56-4c41-8d37-98cd53e9d4e6","Type":"ContainerStarted","Data":"85a60310746f6da7a28182e7f47acfb6dcd48e1be187e699fd8b135ba04240fa"}
Sep 10 19:12:27.529881 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:27.529458    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:12:28.536330 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:28.536285    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-htz6q" event={"ID":"bfc622a4-f58d-42ec-97f6-5271a1a9d3bc","Type":"ContainerStarted","Data":"bcf09e08d149348eab4ba3b64a9f91d1478dfb644578d588f4ce74a2ba27cb65"}
Sep 10 19:12:28.559681 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:28.559620    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-htz6q" podStartSLOduration=76.162073741 podStartE2EDuration="1m23.559602945s" totalImagesPullingTime="7.397529204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:28.558712031 +0000 UTC m=+84.218509226" watchObservedRunningTime="2026-09-10 19:12:28.559602945 +0000 UTC m=+84.219400138"
Sep 10 19:12:28.559859 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:28.559791    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qlmln" podStartSLOduration=76.173506132 podStartE2EDuration="1m23.559782433s" totalImagesPullingTime="7.386276301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:27.556416284 +0000 UTC m=+83.216213479" watchObservedRunningTime="2026-09-10 19:12:28.559782433 +0000 UTC m=+84.219579626"
Sep 10 19:12:44.911369 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:44.911332    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:44.921286 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:44.921229    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-59597c4497-bl8ch"
Sep 10 19:12:47.529957 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.529890    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bf9677f97-4wng5" podUID="b7d6aa9b-6612-4b75-aa89-510966e3a4ae" containerName="console" containerID="cri-o://d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a" gracePeriod=15
Sep 10 19:12:47.816647 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.816603    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:47.850508 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.850477    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-serving-cert\") pod \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") "
Sep 10 19:12:47.850713 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.850520    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-service-ca\") pod \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") "
Sep 10 19:12:47.850713 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.850589    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-oauth-config\") pod \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") "
Sep 10 19:12:47.850713 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.850626    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2gnc\" (UniqueName: \"kubernetes.io/projected/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-kube-api-access-j2gnc\") pod \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") "
Sep 10 19:12:47.850992 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.850965    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-config\") pod \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") "
Sep 10 19:12:47.850992 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.850980    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7d6aa9b-6612-4b75-aa89-510966e3a4ae" (UID: "b7d6aa9b-6612-4b75-aa89-510966e3a4ae"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:12:47.851233 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.851086    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-oauth-serving-cert\") pod \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\" (UID: \"b7d6aa9b-6612-4b75-aa89-510966e3a4ae\") "
Sep 10 19:12:47.851458 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.851429    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-config" (OuterVolumeSpecName: "console-config") pod "b7d6aa9b-6612-4b75-aa89-510966e3a4ae" (UID: "b7d6aa9b-6612-4b75-aa89-510966e3a4ae"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:12:47.851621 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.851559    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-config\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:12:47.851621 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.851577    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:12:47.851746 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.851705    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7d6aa9b-6612-4b75-aa89-510966e3a4ae" (UID: "b7d6aa9b-6612-4b75-aa89-510966e3a4ae"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:12:47.853360 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.853329    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-kube-api-access-j2gnc" (OuterVolumeSpecName: "kube-api-access-j2gnc") pod "b7d6aa9b-6612-4b75-aa89-510966e3a4ae" (UID: "b7d6aa9b-6612-4b75-aa89-510966e3a4ae"). InnerVolumeSpecName "kube-api-access-j2gnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:12:47.853501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.853472    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7d6aa9b-6612-4b75-aa89-510966e3a4ae" (UID: "b7d6aa9b-6612-4b75-aa89-510966e3a4ae"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:12:47.853591 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.853569    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7d6aa9b-6612-4b75-aa89-510966e3a4ae" (UID: "b7d6aa9b-6612-4b75-aa89-510966e3a4ae"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:12:47.952610 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.952497    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2gnc\" (UniqueName: \"kubernetes.io/projected/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-kube-api-access-j2gnc\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:12:47.952610 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.952536    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-oauth-serving-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:12:47.952610 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.952552    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-serving-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:12:47.952610 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:47.952565    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7d6aa9b-6612-4b75-aa89-510966e3a4ae-console-oauth-config\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:12:48.633673 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.633636    2573 generic.go:356] "Generic (PLEG): container finished" podID="b7d6aa9b-6612-4b75-aa89-510966e3a4ae" containerID="d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a" exitCode=0
Sep 10 19:12:48.634195 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.633690    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf9677f97-4wng5" event={"ID":"b7d6aa9b-6612-4b75-aa89-510966e3a4ae","Type":"ContainerDied","Data":"d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a"}
Sep 10 19:12:48.634195 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.633697    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf9677f97-4wng5"
Sep 10 19:12:48.634195 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.633753    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf9677f97-4wng5" event={"ID":"b7d6aa9b-6612-4b75-aa89-510966e3a4ae","Type":"ContainerDied","Data":"d0ad3df2890d0ad2c4ab85d7dd07532434da49ebf4f0b3d89af067dfd001735b"}
Sep 10 19:12:48.634195 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.633773    2573 scope.go:117] "RemoveContainer" containerID="d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a"
Sep 10 19:12:48.646091 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.646075    2573 scope.go:117] "RemoveContainer" containerID="d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a"
Sep 10 19:12:48.646401 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:12:48.646380    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a\": container with ID starting with d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a not found: ID does not exist" containerID="d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a"
Sep 10 19:12:48.646456 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.646410    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a"} err="failed to get container status \"d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a\": rpc error: code = NotFound desc = could not find container \"d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a\": container with ID starting with d34a514f0a9681dca294234301bf32eea156c82ae91c8f7002b75e9833d7779a not found: ID does not exist"
Sep 10 19:12:48.655813 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.655786    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf9677f97-4wng5"]
Sep 10 19:12:48.659722 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.659703    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf9677f97-4wng5"]
Sep 10 19:12:48.832885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:48.832852    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d6aa9b-6612-4b75-aa89-510966e3a4ae" path="/var/lib/kubelet/pods/b7d6aa9b-6612-4b75-aa89-510966e3a4ae/volumes"
Sep 10 19:12:58.539064 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:12:58.539035    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qlmln"
Sep 10 19:13:06.966559 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:06.966517    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:07.029712 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:07.029685    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:07.726519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:07.726491    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:20.709354 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:20.709316    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 19:13:20.709883 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:20.709626    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="alertmanager" containerID="cri-o://03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1" gracePeriod=120
Sep 10 19:13:20.709883 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:20.709644    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="prom-label-proxy" containerID="cri-o://ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035" gracePeriod=120
Sep 10 19:13:20.709883 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:20.709676    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy-web" containerID="cri-o://657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb" gracePeriod=120
Sep 10 19:13:20.709883 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:20.709674    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy" containerID="cri-o://990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac" gracePeriod=120
Sep 10 19:13:20.709883 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:20.709737    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy-metric" containerID="cri-o://02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162" gracePeriod=120
Sep 10 19:13:20.709883 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:20.709751    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="config-reloader" containerID="cri-o://e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42" gracePeriod=120
Sep 10 19:13:21.769793 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769764    2573 generic.go:356] "Generic (PLEG): container finished" podID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerID="ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035" exitCode=0
Sep 10 19:13:21.769793 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769787    2573 generic.go:356] "Generic (PLEG): container finished" podID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerID="02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162" exitCode=0
Sep 10 19:13:21.769793 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769793    2573 generic.go:356] "Generic (PLEG): container finished" podID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerID="990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac" exitCode=0
Sep 10 19:13:21.769793 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769801    2573 generic.go:356] "Generic (PLEG): container finished" podID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerID="e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42" exitCode=0
Sep 10 19:13:21.770341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769807    2573 generic.go:356] "Generic (PLEG): container finished" podID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerID="03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1" exitCode=0
Sep 10 19:13:21.770341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769846    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerDied","Data":"ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035"}
Sep 10 19:13:21.770341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769912    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerDied","Data":"02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162"}
Sep 10 19:13:21.770341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769939    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerDied","Data":"990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac"}
Sep 10 19:13:21.770341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769965    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerDied","Data":"e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42"}
Sep 10 19:13:21.770341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.769991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerDied","Data":"03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1"}
Sep 10 19:13:21.955741 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:21.955716    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.057057 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057024    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-cluster-tls-config\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057057 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057065    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-out\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057097    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-main-db\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057114    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057134    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-tls-assets\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057175    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxz4n\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-kube-api-access-sxz4n\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057196    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-volume\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057219    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-metrics-client-ca\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057250    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-web\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057291    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057354    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-main-tls\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057791 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057379    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-trusted-ca-bundle\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.057791 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057400    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-web-config\") pod \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\" (UID: \"ca06c060-a6e2-498e-bbe8-dd81f7f3194b\") "
Sep 10 19:13:22.058033 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.057936    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:13:22.059101 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.058726    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:13:22.059101 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.059055    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:13:22.061680 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.061122    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:22.062049 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.061770    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:22.063092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.062998    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:22.063092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.063067    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:13:22.063500 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.063456    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-volume" (OuterVolumeSpecName: "config-volume") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:22.063615 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.063594    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-kube-api-access-sxz4n" (OuterVolumeSpecName: "kube-api-access-sxz4n") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "kube-api-access-sxz4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:13:22.063996 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.063969    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-out" (OuterVolumeSpecName: "config-out") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:13:22.064331 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.064303    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:22.068962 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.068934    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:22.075916 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.075889    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-web-config" (OuterVolumeSpecName: "web-config") pod "ca06c060-a6e2-498e-bbe8-dd81f7f3194b" (UID: "ca06c060-a6e2-498e-bbe8-dd81f7f3194b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:22.158828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158784    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-main-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.158828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158818    2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.158828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158828    2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-web-config\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.158828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158839    2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-cluster-tls-config\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.159088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158848    2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-out\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.159088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158855    2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-alertmanager-main-db\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.159088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158863    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.159088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158872    2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-tls-assets\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.159088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158880    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxz4n\" (UniqueName: \"kubernetes.io/projected/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-kube-api-access-sxz4n\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.159088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158889    2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.159088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158897    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-metrics-client-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.159088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158906    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.159088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.158915    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ca06c060-a6e2-498e-bbe8-dd81f7f3194b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:22.777347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.777309    2573 generic.go:356] "Generic (PLEG): container finished" podID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerID="657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb" exitCode=0
Sep 10 19:13:22.777711 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.777389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerDied","Data":"657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb"}
Sep 10 19:13:22.777711 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.777464    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ca06c060-a6e2-498e-bbe8-dd81f7f3194b","Type":"ContainerDied","Data":"4a4dceae010dcbf00693e992a0a83c2a5e1fda8c10a995eae83e9de9e9e49984"}
Sep 10 19:13:22.777711 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.777402    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.777711 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.777510    2573 scope.go:117] "RemoveContainer" containerID="ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035"
Sep 10 19:13:22.791680 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.791660    2573 scope.go:117] "RemoveContainer" containerID="02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162"
Sep 10 19:13:22.799372 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.799351    2573 scope.go:117] "RemoveContainer" containerID="990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac"
Sep 10 19:13:22.807627 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.807588    2573 scope.go:117] "RemoveContainer" containerID="657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb"
Sep 10 19:13:22.809124 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.809104    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 19:13:22.815165 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.815129    2573 scope.go:117] "RemoveContainer" containerID="e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42"
Sep 10 19:13:22.822074 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.822020    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 19:13:22.826891 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.826759    2573 scope.go:117] "RemoveContainer" containerID="03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1"
Sep 10 19:13:22.832357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.832326    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" path="/var/lib/kubelet/pods/ca06c060-a6e2-498e-bbe8-dd81f7f3194b/volumes"
Sep 10 19:13:22.835256 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.835240    2573 scope.go:117] "RemoveContainer" containerID="1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec"
Sep 10 19:13:22.842711 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.842692    2573 scope.go:117] "RemoveContainer" containerID="ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035"
Sep 10 19:13:22.843009 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:22.842990    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035\": container with ID starting with ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035 not found: ID does not exist" containerID="ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035"
Sep 10 19:13:22.843083 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.843021    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035"} err="failed to get container status \"ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035\": rpc error: code = NotFound desc = could not find container \"ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035\": container with ID starting with ede9499bd3cea87d09a4529628c7844a2e3ccaeaa5eb48e8b90ca2fda8cbf035 not found: ID does not exist"
Sep 10 19:13:22.843083 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.843048    2573 scope.go:117] "RemoveContainer" containerID="02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162"
Sep 10 19:13:22.843332 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:22.843314    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162\": container with ID starting with 02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162 not found: ID does not exist" containerID="02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162"
Sep 10 19:13:22.843388 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.843339    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162"} err="failed to get container status \"02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162\": rpc error: code = NotFound desc = could not find container \"02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162\": container with ID starting with 02dbb317e3eb243dce10a9a3a945a49616b2fa8c67c1f4b7242645af5c056162 not found: ID does not exist"
Sep 10 19:13:22.843388 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.843354    2573 scope.go:117] "RemoveContainer" containerID="990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac"
Sep 10 19:13:22.843613 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:22.843597    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac\": container with ID starting with 990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac not found: ID does not exist" containerID="990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac"
Sep 10 19:13:22.843655 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.843616    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac"} err="failed to get container status \"990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac\": rpc error: code = NotFound desc = could not find container \"990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac\": container with ID starting with 990c7b3b7c3337d30fe5257ff702cba8bcb187a94122c900e2d9be760e788bac not found: ID does not exist"
Sep 10 19:13:22.843655 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.843629    2573 scope.go:117] "RemoveContainer" containerID="657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb"
Sep 10 19:13:22.843875 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:22.843859    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb\": container with ID starting with 657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb not found: ID does not exist" containerID="657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb"
Sep 10 19:13:22.843933 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.843883    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb"} err="failed to get container status \"657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb\": rpc error: code = NotFound desc = could not find container \"657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb\": container with ID starting with 657a8bda1f0a6e531bb07b36cdbd85a248d9fd8ce89f15507136cd92e4e79dcb not found: ID does not exist"
Sep 10 19:13:22.843933 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.843906    2573 scope.go:117] "RemoveContainer" containerID="e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42"
Sep 10 19:13:22.844143 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:22.844128    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42\": container with ID starting with e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42 not found: ID does not exist" containerID="e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42"
Sep 10 19:13:22.844217 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.844162    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42"} err="failed to get container status \"e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42\": rpc error: code = NotFound desc = could not find container \"e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42\": container with ID starting with e8dc1b084d64e58efb523d8df840491b5f6424a6127834111dbefd9874f7bb42 not found: ID does not exist"
Sep 10 19:13:22.844217 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.844177    2573 scope.go:117] "RemoveContainer" containerID="03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1"
Sep 10 19:13:22.844398 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:22.844381    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1\": container with ID starting with 03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1 not found: ID does not exist" containerID="03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1"
Sep 10 19:13:22.844436 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.844401    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1"} err="failed to get container status \"03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1\": rpc error: code = NotFound desc = could not find container \"03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1\": container with ID starting with 03e66c95173b29efb4934ea82990662372884ae172bd70ca8157c24ee31ef4c1 not found: ID does not exist"
Sep 10 19:13:22.844436 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.844413    2573 scope.go:117] "RemoveContainer" containerID="1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec"
Sep 10 19:13:22.844610 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:22.844595    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec\": container with ID starting with 1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec not found: ID does not exist" containerID="1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec"
Sep 10 19:13:22.844652 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.844612    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec"} err="failed to get container status \"1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec\": rpc error: code = NotFound desc = could not find container \"1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec\": container with ID starting with 1548abddde1b14e6964332b5370c035542f8b8e95c9aa283813ebc61f5466aec not found: ID does not exist"
Sep 10 19:13:22.863771 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.863740    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 19:13:22.864092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864079    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d6aa9b-6612-4b75-aa89-510966e3a4ae" containerName="console"
Sep 10 19:13:22.864135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864094    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d6aa9b-6612-4b75-aa89-510966e3a4ae" containerName="console"
Sep 10 19:13:22.864135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864104    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="prom-label-proxy"
Sep 10 19:13:22.864135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864110    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="prom-label-proxy"
Sep 10 19:13:22.864135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864129    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="init-config-reloader"
Sep 10 19:13:22.864135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864135    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="init-config-reloader"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864142    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy-web"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864165    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy-web"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864175    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864181    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864193    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy-metric"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864197    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy-metric"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864205    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="config-reloader"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864211    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="config-reloader"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864222    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="alertmanager"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864227    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="alertmanager"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864278    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="prom-label-proxy"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864287    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="config-reloader"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864295    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy-metric"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864302    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy-web"
Sep 10 19:13:22.864307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864307    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d6aa9b-6612-4b75-aa89-510966e3a4ae" containerName="console"
Sep 10 19:13:22.864704 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864315    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="alertmanager"
Sep 10 19:13:22.864704 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.864324    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca06c060-a6e2-498e-bbe8-dd81f7f3194b" containerName="kube-rbac-proxy"
Sep 10 19:13:22.867734 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.867719    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.872717 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.872212    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 19:13:22.872717 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.872341    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vjllq\""
Sep 10 19:13:22.872717 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.872390    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 19:13:22.872717 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.872426    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 19:13:22.872717 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.872516    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 19:13:22.872717 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.872624    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 19:13:22.873651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.873427    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 19:13:22.873651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.873457    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 19:13:22.873907 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.873890    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 19:13:22.887634 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.887613    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 19:13:22.899076 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.899050    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 19:13:22.967632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967596    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.967632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d061bc22-f892-4d24-8d95-857ff39d1ed3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.967900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d061bc22-f892-4d24-8d95-857ff39d1ed3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.967900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967672    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d061bc22-f892-4d24-8d95-857ff39d1ed3-config-out\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.967900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967755    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.967900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.967900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967817    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.967900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.967900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d061bc22-f892-4d24-8d95-857ff39d1ed3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.968134 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967907    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-config-volume\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.968134 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967934    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-web-config\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.968134 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.967984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d061bc22-f892-4d24-8d95-857ff39d1ed3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:22.968134 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:22.968016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f62nj\" (UniqueName: \"kubernetes.io/projected/d061bc22-f892-4d24-8d95-857ff39d1ed3-kube-api-access-f62nj\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.068713 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.068635    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d061bc22-f892-4d24-8d95-857ff39d1ed3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.068713 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.068675    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d061bc22-f892-4d24-8d95-857ff39d1ed3-config-out\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.068713 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.068698    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.068922 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.068723    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.068922 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.068748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.068922 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.068884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.069053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.068933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d061bc22-f892-4d24-8d95-857ff39d1ed3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.069053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.068974    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-config-volume\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.069053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.069008    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-web-config\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.069237 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.069054    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d061bc22-f892-4d24-8d95-857ff39d1ed3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.069237 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.069121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f62nj\" (UniqueName: \"kubernetes.io/projected/d061bc22-f892-4d24-8d95-857ff39d1ed3-kube-api-access-f62nj\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.069237 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.069222    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.069384 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.069261    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d061bc22-f892-4d24-8d95-857ff39d1ed3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.069580 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.069549    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d061bc22-f892-4d24-8d95-857ff39d1ed3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.069794 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.069754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d061bc22-f892-4d24-8d95-857ff39d1ed3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.070445 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.070253    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d061bc22-f892-4d24-8d95-857ff39d1ed3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.071783 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.071756    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d061bc22-f892-4d24-8d95-857ff39d1ed3-config-out\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.072024 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.071978    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.072368 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.072344    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.072890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.072771    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.072890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.072835    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-web-config\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.072890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.072883    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-config-volume\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.073181 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.073138    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.073277 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.073261    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d061bc22-f892-4d24-8d95-857ff39d1ed3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.073742 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.073722    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d061bc22-f892-4d24-8d95-857ff39d1ed3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.084580 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.084556    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f62nj\" (UniqueName: \"kubernetes.io/projected/d061bc22-f892-4d24-8d95-857ff39d1ed3-kube-api-access-f62nj\") pod \"alertmanager-main-0\" (UID: \"d061bc22-f892-4d24-8d95-857ff39d1ed3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.178393 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.178353    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 19:13:23.464093 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.464020    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 19:13:23.467329 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:13:23.467290    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd061bc22_f892_4d24_8d95_857ff39d1ed3.slice/crio-ccc2680ecd141aed1bc35aefe4df004362afc97efbc9ef82139d622a2de6490e WatchSource:0}: Error finding container ccc2680ecd141aed1bc35aefe4df004362afc97efbc9ef82139d622a2de6490e: Status 404 returned error can't find the container with id ccc2680ecd141aed1bc35aefe4df004362afc97efbc9ef82139d622a2de6490e
Sep 10 19:13:23.783862 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.783819    2573 generic.go:356] "Generic (PLEG): container finished" podID="d061bc22-f892-4d24-8d95-857ff39d1ed3" containerID="ea3e69a695235ebf0a9ddb37e1a01126b9c62eb4bb539aab133b9b36641aba1f" exitCode=0
Sep 10 19:13:23.784269 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.783915    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d061bc22-f892-4d24-8d95-857ff39d1ed3","Type":"ContainerDied","Data":"ea3e69a695235ebf0a9ddb37e1a01126b9c62eb4bb539aab133b9b36641aba1f"}
Sep 10 19:13:23.784269 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:23.783982    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d061bc22-f892-4d24-8d95-857ff39d1ed3","Type":"ContainerStarted","Data":"ccc2680ecd141aed1bc35aefe4df004362afc97efbc9ef82139d622a2de6490e"}
Sep 10 19:13:24.792521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:24.792485    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d061bc22-f892-4d24-8d95-857ff39d1ed3","Type":"ContainerStarted","Data":"0ae3e70107fd247acdad6e67b9631e19a168ee89d16bd081bf79633f261311bd"}
Sep 10 19:13:24.792906 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:24.792566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d061bc22-f892-4d24-8d95-857ff39d1ed3","Type":"ContainerStarted","Data":"91bb899e7a2675a822ca3881409199a1584800c08d5c160fe65042c6688170b3"}
Sep 10 19:13:24.792906 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:24.792613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d061bc22-f892-4d24-8d95-857ff39d1ed3","Type":"ContainerStarted","Data":"5505327b77d94ae40d1dc05f24cf4ddaebaaf25d6c71d025cec3fd0d7bde8e82"}
Sep 10 19:13:24.792906 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:24.792648    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d061bc22-f892-4d24-8d95-857ff39d1ed3","Type":"ContainerStarted","Data":"ee170faf9b384cdde5672396275027c1ac31ad3e74c111f8946e4e37f14a87c3"}
Sep 10 19:13:24.792906 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:24.792683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d061bc22-f892-4d24-8d95-857ff39d1ed3","Type":"ContainerStarted","Data":"fcf948056737168d0b2e4c223914b4abb156cba7c8334e6c4e6435dbc959effe"}
Sep 10 19:13:24.792906 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:24.792726    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d061bc22-f892-4d24-8d95-857ff39d1ed3","Type":"ContainerStarted","Data":"25fe9485d562b8b8b3ca9098a893e070abf8b183b8fd2bf27de9945f29ed2911"}
Sep 10 19:13:24.832343 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:24.832223    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.817245209 podStartE2EDuration="2.83219677s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.951561ms" isStatefulPod=false podCreationTimestamp="2026-09-10 19:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:13:24.829609059 +0000 UTC m=+140.489406279" watchObservedRunningTime="2026-09-10 19:13:24.83219677 +0000 UTC m=+140.491993964"
Sep 10 19:13:25.036318 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.036275    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 19:13:25.036671 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.036635    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="prometheus" containerID="cri-o://444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04" gracePeriod=600
Sep 10 19:13:25.037049 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.036998    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="config-reloader" containerID="cri-o://7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f" gracePeriod=600
Sep 10 19:13:25.037137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.037032    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304" gracePeriod=600
Sep 10 19:13:25.037137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.037074    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="thanos-sidecar" containerID="cri-o://98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a" gracePeriod=600
Sep 10 19:13:25.037137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.037079    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy" containerID="cri-o://9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a" gracePeriod=600
Sep 10 19:13:25.037344 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.037285    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy-web" containerID="cri-o://1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db" gracePeriod=600
Sep 10 19:13:25.290675 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.290360    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.390196 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390080    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-serving-certs-ca-bundle\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.390196 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390161    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.390411 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390210    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-db\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.390411 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390236    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-kube-rbac-proxy\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.390411 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390258    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-metrics-client-ca\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.390411 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390283    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-kubelet-serving-ca-bundle\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.390411 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390306    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-metrics-client-certs\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.390673 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390537    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:13:25.390673 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390634    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:13:25.390855 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390815    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-0\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391084 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390956    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:13:25.391084 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.390970    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-config-out\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391084 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391052    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-grpc-tls\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391292 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391090    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-2\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391292 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391198    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-tls\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391292 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391228    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-1\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391292 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391261    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391292 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391289    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-config\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391539 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391319    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjk2r\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-kube-api-access-vjk2r\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391539 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391343    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-web-config\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391539 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391371    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-trusted-ca-bundle\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391539 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391421    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-thanos-prometheus-http-client-file\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391539 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391448    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-tls-assets\") pod \"39748edc-cb63-41e8-b9c0-2c13417157cb\" (UID: \"39748edc-cb63-41e8-b9c0-2c13417157cb\") "
Sep 10 19:13:25.391777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391727    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-metrics-client-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.391777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391745    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.391777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391761    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.393123 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391336    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:13:25.393123 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.391560    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:13:25.393123 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.392413    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:13:25.393123 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.393040    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:13:25.393509 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.393115    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:25.393509 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.393275    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:25.393806 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.393770    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:25.394100 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.394073    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:13:25.394777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.394740    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-config-out" (OuterVolumeSpecName: "config-out") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:13:25.395768 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.395730    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:25.396228 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.396190    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-config" (OuterVolumeSpecName: "config") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:25.396361 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.396225    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-kube-api-access-vjk2r" (OuterVolumeSpecName: "kube-api-access-vjk2r") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "kube-api-access-vjk2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:13:25.396462 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.396440    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:25.396880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.396857    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:25.396984 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.396962    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:13:25.397643 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.397625    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:25.406949 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.406926    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-web-config" (OuterVolumeSpecName: "web-config") pod "39748edc-cb63-41e8-b9c0-2c13417157cb" (UID: "39748edc-cb63-41e8-b9c0-2c13417157cb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:13:25.492741 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492706    2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-kube-rbac-proxy\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492741 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492732    2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-metrics-client-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492741 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492742    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492741 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492751    2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-config-out\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492759    2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-grpc-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492766    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492775    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492783    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492792    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492802    2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-config\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492810    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjk2r\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-kube-api-access-vjk2r\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492819    2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-web-config\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492826    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492835    2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492842    2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39748edc-cb63-41e8-b9c0-2c13417157cb-tls-assets\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492851    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39748edc-cb63-41e8-b9c0-2c13417157cb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.492998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.492860    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39748edc-cb63-41e8-b9c0-2c13417157cb-prometheus-k8s-db\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:13:25.799659 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799618    2573 generic.go:356] "Generic (PLEG): container finished" podID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerID="8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304" exitCode=0
Sep 10 19:13:25.799659 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799650    2573 generic.go:356] "Generic (PLEG): container finished" podID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerID="9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a" exitCode=0
Sep 10 19:13:25.799659 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799661    2573 generic.go:356] "Generic (PLEG): container finished" podID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerID="1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db" exitCode=0
Sep 10 19:13:25.799659 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799670    2573 generic.go:356] "Generic (PLEG): container finished" podID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerID="98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a" exitCode=0
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799679    2573 generic.go:356] "Generic (PLEG): container finished" podID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerID="7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f" exitCode=0
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799687    2573 generic.go:356] "Generic (PLEG): container finished" podID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerID="444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04" exitCode=0
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799715    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799705    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerDied","Data":"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"}
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799895    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerDied","Data":"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"}
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799949    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerDied","Data":"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"}
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799963    2573 scope.go:117] "RemoveContainer" containerID="8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.799992    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerDied","Data":"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"}
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.800027    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerDied","Data":"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"}
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.800062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerDied","Data":"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"}
Sep 10 19:13:25.800309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.800099    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39748edc-cb63-41e8-b9c0-2c13417157cb","Type":"ContainerDied","Data":"2de07de16cbfe86fd8aadbb0d2513c69ba96004c15a76aca37c75b7cbc4b678e"}
Sep 10 19:13:25.808310 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.808289    2573 scope.go:117] "RemoveContainer" containerID="9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"
Sep 10 19:13:25.815879 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.815861    2573 scope.go:117] "RemoveContainer" containerID="1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"
Sep 10 19:13:25.822817 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.822798    2573 scope.go:117] "RemoveContainer" containerID="98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"
Sep 10 19:13:25.826930 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.826907    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 19:13:25.830302 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.830283    2573 scope.go:117] "RemoveContainer" containerID="7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"
Sep 10 19:13:25.835892 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.835868    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 19:13:25.838405 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.838390    2573 scope.go:117] "RemoveContainer" containerID="444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"
Sep 10 19:13:25.847943 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.847921    2573 scope.go:117] "RemoveContainer" containerID="bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"
Sep 10 19:13:25.855189 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.855170    2573 scope.go:117] "RemoveContainer" containerID="8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"
Sep 10 19:13:25.855476 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:25.855457    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": container with ID starting with 8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304 not found: ID does not exist" containerID="8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"
Sep 10 19:13:25.855541 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.855487    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"} err="failed to get container status \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": rpc error: code = NotFound desc = could not find container \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": container with ID starting with 8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304 not found: ID does not exist"
Sep 10 19:13:25.855541 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.855508    2573 scope.go:117] "RemoveContainer" containerID="9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"
Sep 10 19:13:25.855860 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:25.855834    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": container with ID starting with 9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a not found: ID does not exist" containerID="9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"
Sep 10 19:13:25.855926 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.855866    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"} err="failed to get container status \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": rpc error: code = NotFound desc = could not find container \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": container with ID starting with 9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a not found: ID does not exist"
Sep 10 19:13:25.855926 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.855884    2573 scope.go:117] "RemoveContainer" containerID="1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"
Sep 10 19:13:25.856284 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:25.856265    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": container with ID starting with 1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db not found: ID does not exist" containerID="1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"
Sep 10 19:13:25.856359 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.856289    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"} err="failed to get container status \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": rpc error: code = NotFound desc = could not find container \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": container with ID starting with 1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db not found: ID does not exist"
Sep 10 19:13:25.856359 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.856302    2573 scope.go:117] "RemoveContainer" containerID="98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"
Sep 10 19:13:25.856644 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:25.856627    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": container with ID starting with 98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a not found: ID does not exist" containerID="98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"
Sep 10 19:13:25.856686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.856654    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"} err="failed to get container status \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": rpc error: code = NotFound desc = could not find container \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": container with ID starting with 98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a not found: ID does not exist"
Sep 10 19:13:25.856686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.856671    2573 scope.go:117] "RemoveContainer" containerID="7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"
Sep 10 19:13:25.856953 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:25.856938    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": container with ID starting with 7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f not found: ID does not exist" containerID="7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"
Sep 10 19:13:25.856994 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.856956    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"} err="failed to get container status \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": rpc error: code = NotFound desc = could not find container \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": container with ID starting with 7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f not found: ID does not exist"
Sep 10 19:13:25.856994 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.856974    2573 scope.go:117] "RemoveContainer" containerID="444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"
Sep 10 19:13:25.857216 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:25.857201    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": container with ID starting with 444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04 not found: ID does not exist" containerID="444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"
Sep 10 19:13:25.857265 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.857218    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"} err="failed to get container status \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": rpc error: code = NotFound desc = could not find container \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": container with ID starting with 444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04 not found: ID does not exist"
Sep 10 19:13:25.857265 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.857230    2573 scope.go:117] "RemoveContainer" containerID="bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"
Sep 10 19:13:25.857460 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:13:25.857447    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": container with ID starting with bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf not found: ID does not exist" containerID="bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"
Sep 10 19:13:25.857497 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.857463    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"} err="failed to get container status \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": rpc error: code = NotFound desc = could not find container \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": container with ID starting with bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf not found: ID does not exist"
Sep 10 19:13:25.857497 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.857480    2573 scope.go:117] "RemoveContainer" containerID="8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"
Sep 10 19:13:25.857745 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.857726    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"} err="failed to get container status \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": rpc error: code = NotFound desc = could not find container \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": container with ID starting with 8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304 not found: ID does not exist"
Sep 10 19:13:25.857798 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.857745    2573 scope.go:117] "RemoveContainer" containerID="9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"
Sep 10 19:13:25.857968 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.857950    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"} err="failed to get container status \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": rpc error: code = NotFound desc = could not find container \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": container with ID starting with 9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a not found: ID does not exist"
Sep 10 19:13:25.858016 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.857968    2573 scope.go:117] "RemoveContainer" containerID="1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"
Sep 10 19:13:25.858212 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.858193    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"} err="failed to get container status \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": rpc error: code = NotFound desc = could not find container \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": container with ID starting with 1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db not found: ID does not exist"
Sep 10 19:13:25.858292 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.858215    2573 scope.go:117] "RemoveContainer" containerID="98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"
Sep 10 19:13:25.858485 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.858463    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"} err="failed to get container status \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": rpc error: code = NotFound desc = could not find container \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": container with ID starting with 98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a not found: ID does not exist"
Sep 10 19:13:25.858485 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.858483    2573 scope.go:117] "RemoveContainer" containerID="7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"
Sep 10 19:13:25.858727 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.858710    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"} err="failed to get container status \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": rpc error: code = NotFound desc = could not find container \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": container with ID starting with 7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f not found: ID does not exist"
Sep 10 19:13:25.858775 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.858727    2573 scope.go:117] "RemoveContainer" containerID="444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"
Sep 10 19:13:25.858971 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.858945    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"} err="failed to get container status \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": rpc error: code = NotFound desc = could not find container \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": container with ID starting with 444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04 not found: ID does not exist"
Sep 10 19:13:25.859046 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.858973    2573 scope.go:117] "RemoveContainer" containerID="bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"
Sep 10 19:13:25.859219 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.859202    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"} err="failed to get container status \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": rpc error: code = NotFound desc = could not find container \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": container with ID starting with bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf not found: ID does not exist"
Sep 10 19:13:25.859282 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.859221    2573 scope.go:117] "RemoveContainer" containerID="8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"
Sep 10 19:13:25.859447 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.859427    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"} err="failed to get container status \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": rpc error: code = NotFound desc = could not find container \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": container with ID starting with 8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304 not found: ID does not exist"
Sep 10 19:13:25.859507 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.859449    2573 scope.go:117] "RemoveContainer" containerID="9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"
Sep 10 19:13:25.859695 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.859678    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"} err="failed to get container status \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": rpc error: code = NotFound desc = could not find container \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": container with ID starting with 9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a not found: ID does not exist"
Sep 10 19:13:25.859758 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.859699    2573 scope.go:117] "RemoveContainer" containerID="1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"
Sep 10 19:13:25.859975 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.859957    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"} err="failed to get container status \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": rpc error: code = NotFound desc = could not find container \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": container with ID starting with 1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db not found: ID does not exist"
Sep 10 19:13:25.860022 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.859977    2573 scope.go:117] "RemoveContainer" containerID="98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"
Sep 10 19:13:25.860229 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.860213    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"} err="failed to get container status \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": rpc error: code = NotFound desc = could not find container \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": container with ID starting with 98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a not found: ID does not exist"
Sep 10 19:13:25.860282 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.860228    2573 scope.go:117] "RemoveContainer" containerID="7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"
Sep 10 19:13:25.860422 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.860402    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"} err="failed to get container status \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": rpc error: code = NotFound desc = could not find container \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": container with ID starting with 7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f not found: ID does not exist"
Sep 10 19:13:25.860485 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.860425    2573 scope.go:117] "RemoveContainer" containerID="444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"
Sep 10 19:13:25.860647 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.860621    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"} err="failed to get container status \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": rpc error: code = NotFound desc = could not find container \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": container with ID starting with 444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04 not found: ID does not exist"
Sep 10 19:13:25.860704 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.860649    2573 scope.go:117] "RemoveContainer" containerID="bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"
Sep 10 19:13:25.860887 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.860871    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"} err="failed to get container status \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": rpc error: code = NotFound desc = could not find container \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": container with ID starting with bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf not found: ID does not exist"
Sep 10 19:13:25.860944 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.860888    2573 scope.go:117] "RemoveContainer" containerID="8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"
Sep 10 19:13:25.861077 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.861060    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"} err="failed to get container status \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": rpc error: code = NotFound desc = could not find container \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": container with ID starting with 8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304 not found: ID does not exist"
Sep 10 19:13:25.861144 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.861079    2573 scope.go:117] "RemoveContainer" containerID="9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"
Sep 10 19:13:25.861361 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.861342    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"} err="failed to get container status \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": rpc error: code = NotFound desc = could not find container \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": container with ID starting with 9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a not found: ID does not exist"
Sep 10 19:13:25.861423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.861362    2573 scope.go:117] "RemoveContainer" containerID="1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"
Sep 10 19:13:25.861569 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.861553    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"} err="failed to get container status \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": rpc error: code = NotFound desc = could not find container \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": container with ID starting with 1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db not found: ID does not exist"
Sep 10 19:13:25.861637 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.861571    2573 scope.go:117] "RemoveContainer" containerID="98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"
Sep 10 19:13:25.861806 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.861789    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"} err="failed to get container status \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": rpc error: code = NotFound desc = could not find container \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": container with ID starting with 98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a not found: ID does not exist"
Sep 10 19:13:25.861866 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.861807    2573 scope.go:117] "RemoveContainer" containerID="7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"
Sep 10 19:13:25.862036 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862019    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"} err="failed to get container status \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": rpc error: code = NotFound desc = could not find container \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": container with ID starting with 7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f not found: ID does not exist"
Sep 10 19:13:25.862081 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862037    2573 scope.go:117] "RemoveContainer" containerID="444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"
Sep 10 19:13:25.862290 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862272    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"} err="failed to get container status \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": rpc error: code = NotFound desc = could not find container \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": container with ID starting with 444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04 not found: ID does not exist"
Sep 10 19:13:25.862345 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862291    2573 scope.go:117] "RemoveContainer" containerID="bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"
Sep 10 19:13:25.862532 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862511    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"} err="failed to get container status \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": rpc error: code = NotFound desc = could not find container \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": container with ID starting with bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf not found: ID does not exist"
Sep 10 19:13:25.862532 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862532    2573 scope.go:117] "RemoveContainer" containerID="8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"
Sep 10 19:13:25.862740 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862724    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"} err="failed to get container status \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": rpc error: code = NotFound desc = could not find container \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": container with ID starting with 8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304 not found: ID does not exist"
Sep 10 19:13:25.862740 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862740    2573 scope.go:117] "RemoveContainer" containerID="9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"
Sep 10 19:13:25.862933 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862918    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"} err="failed to get container status \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": rpc error: code = NotFound desc = could not find container \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": container with ID starting with 9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a not found: ID does not exist"
Sep 10 19:13:25.862988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.862933    2573 scope.go:117] "RemoveContainer" containerID="1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"
Sep 10 19:13:25.863125 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863110    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"} err="failed to get container status \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": rpc error: code = NotFound desc = could not find container \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": container with ID starting with 1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db not found: ID does not exist"
Sep 10 19:13:25.863206 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863125    2573 scope.go:117] "RemoveContainer" containerID="98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"
Sep 10 19:13:25.863356 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863339    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"} err="failed to get container status \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": rpc error: code = NotFound desc = could not find container \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": container with ID starting with 98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a not found: ID does not exist"
Sep 10 19:13:25.863356 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863356    2573 scope.go:117] "RemoveContainer" containerID="7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"
Sep 10 19:13:25.863545 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863529    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"} err="failed to get container status \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": rpc error: code = NotFound desc = could not find container \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": container with ID starting with 7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f not found: ID does not exist"
Sep 10 19:13:25.863600 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863545    2573 scope.go:117] "RemoveContainer" containerID="444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"
Sep 10 19:13:25.863745 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863729    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"} err="failed to get container status \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": rpc error: code = NotFound desc = could not find container \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": container with ID starting with 444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04 not found: ID does not exist"
Sep 10 19:13:25.863745 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863744    2573 scope.go:117] "RemoveContainer" containerID="bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"
Sep 10 19:13:25.863961 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863945    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"} err="failed to get container status \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": rpc error: code = NotFound desc = could not find container \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": container with ID starting with bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf not found: ID does not exist"
Sep 10 19:13:25.864007 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.863961    2573 scope.go:117] "RemoveContainer" containerID="8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"
Sep 10 19:13:25.864220 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.864201    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304"} err="failed to get container status \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": rpc error: code = NotFound desc = could not find container \"8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304\": container with ID starting with 8b452fe09f33221195386bca8a3f7a24be64803e4023fb91f18650758d240304 not found: ID does not exist"
Sep 10 19:13:25.864270 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.864220    2573 scope.go:117] "RemoveContainer" containerID="9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"
Sep 10 19:13:25.864428 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.864409    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a"} err="failed to get container status \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": rpc error: code = NotFound desc = could not find container \"9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a\": container with ID starting with 9b40833ebf6439f47946791465c2d105fda2caf46df222d4b9deebafda94760a not found: ID does not exist"
Sep 10 19:13:25.864488 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.864430    2573 scope.go:117] "RemoveContainer" containerID="1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"
Sep 10 19:13:25.864646 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.864630    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db"} err="failed to get container status \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": rpc error: code = NotFound desc = could not find container \"1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db\": container with ID starting with 1433b4c96283c30c1e93c5463265c9a74d0032aad0eeabe6a90b8e02ed8856db not found: ID does not exist"
Sep 10 19:13:25.864692 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.864646    2573 scope.go:117] "RemoveContainer" containerID="98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"
Sep 10 19:13:25.864845 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.864828    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a"} err="failed to get container status \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": rpc error: code = NotFound desc = could not find container \"98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a\": container with ID starting with 98cbfcba7616d96ae1d17995e82bc1650b0b1e9dde24293546acf2605b62c49a not found: ID does not exist"
Sep 10 19:13:25.864894 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.864845    2573 scope.go:117] "RemoveContainer" containerID="7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"
Sep 10 19:13:25.865038 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.865022    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f"} err="failed to get container status \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": rpc error: code = NotFound desc = could not find container \"7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f\": container with ID starting with 7f94b69b40c644a39c6db1945344371140c84ff8081b1455b2956e26e4a4fb1f not found: ID does not exist"
Sep 10 19:13:25.865087 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.865038    2573 scope.go:117] "RemoveContainer" containerID="444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"
Sep 10 19:13:25.865300 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.865283    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04"} err="failed to get container status \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": rpc error: code = NotFound desc = could not find container \"444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04\": container with ID starting with 444920d708c1cd62cfe5107e5b5bf94ed2c170452e092f5209e481b88370ac04 not found: ID does not exist"
Sep 10 19:13:25.865390 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.865301    2573 scope.go:117] "RemoveContainer" containerID="bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"
Sep 10 19:13:25.865598 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.865578    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf"} err="failed to get container status \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": rpc error: code = NotFound desc = could not find container \"bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf\": container with ID starting with bc9009fa674c7953f451c7934225d72ccf319aaa42e124d836700e011ef3abaf not found: ID does not exist"
Sep 10 19:13:25.882204 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882169    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 19:13:25.882501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882488    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="init-config-reloader"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882503    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="init-config-reloader"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882511    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="prometheus"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882518    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="prometheus"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882523    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy-thanos"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882529    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy-thanos"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882536    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882541    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882551    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="config-reloader"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882556    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="config-reloader"
Sep 10 19:13:25.882561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882563    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="thanos-sidecar"
Sep 10 19:13:25.882885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882569    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="thanos-sidecar"
Sep 10 19:13:25.882885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882576    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy-web"
Sep 10 19:13:25.882885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882581    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy-web"
Sep 10 19:13:25.882885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882623    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="prometheus"
Sep 10 19:13:25.882885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882631    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy-thanos"
Sep 10 19:13:25.882885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882639    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="config-reloader"
Sep 10 19:13:25.882885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882646    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy-web"
Sep 10 19:13:25.882885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882651    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="kube-rbac-proxy"
Sep 10 19:13:25.882885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.882656    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" containerName="thanos-sidecar"
Sep 10 19:13:25.888075 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.888053    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.895331 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.895301    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 19:13:25.895331 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.895327    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 19:13:25.895500 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.895336    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 19:13:25.895994 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.895973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896058 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896011    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65193eb1-624a-4625-ab5b-eb6068942804-config-out\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896109 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896084    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 19:13:25.896184 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896115    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896246 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896175    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-web-config\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896246 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896214    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896346 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896245    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 19:13:25.896346 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896243    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-config\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896346 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896265    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 19:13:25.896346 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896279    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896346 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896303    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8qxc\" (UniqueName: \"kubernetes.io/projected/65193eb1-624a-4625-ab5b-eb6068942804-kube-api-access-w8qxc\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896579 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896358    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65193eb1-624a-4625-ab5b-eb6068942804-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896579 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896493    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896579 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896721 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896645    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896929 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.896929 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.897049 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.896928    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.897049 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897021    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.897129 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.897215 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897126    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.897215 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897182    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.897389 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897374    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 19:13:25.897510 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897493    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 19:13:25.897510 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897505    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 19:13:25.897620 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897506    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 19:13:25.897670 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897648    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 19:13:25.897722 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897668    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5bzmk\""
Sep 10 19:13:25.897722 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.897684    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fog6mbhi4hoic\""
Sep 10 19:13:25.902246 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.902225    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 19:13:25.904966 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.904747    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 19:13:25.909032 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.909013    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 19:13:25.928055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.928026    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 19:13:25.998447 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998447 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998450    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65193eb1-624a-4625-ab5b-eb6068942804-config-out\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-web-config\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998508    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998523    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-config\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998544    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998567    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8qxc\" (UniqueName: \"kubernetes.io/projected/65193eb1-624a-4625-ab5b-eb6068942804-kube-api-access-w8qxc\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998593    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65193eb1-624a-4625-ab5b-eb6068942804-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.998719 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998847    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998915    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.998967    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999925 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.999637    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:25.999925 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:25.999899    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.000056 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.000027    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.000580 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.000550    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.001321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.000965    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.001321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.001204    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.002640 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.001789    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.002640 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.001995    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65193eb1-624a-4625-ab5b-eb6068942804-config-out\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.002640 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.002588    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-config\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.002982 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.002954    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.003458 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.003430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-web-config\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.003802 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.003762    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65193eb1-624a-4625-ab5b-eb6068942804-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.004083 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.004062    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.004307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.004276    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.004422 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.004403    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.004491 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.004470    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.004635 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.004611    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.004999 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.004980    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/65193eb1-624a-4625-ab5b-eb6068942804-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.005714 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.005694    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65193eb1-624a-4625-ab5b-eb6068942804-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.008787 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.008767    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8qxc\" (UniqueName: \"kubernetes.io/projected/65193eb1-624a-4625-ab5b-eb6068942804-kube-api-access-w8qxc\") pod \"prometheus-k8s-0\" (UID: \"65193eb1-624a-4625-ab5b-eb6068942804\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.198380 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.198293    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:13:26.485211 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.485186    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 19:13:26.486401 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:13:26.486371    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65193eb1_624a_4625_ab5b_eb6068942804.slice/crio-a12a6e9a0e20cf2969e165d985ccfc6dfce9d0b15fc8367011da51bf8eb905a3 WatchSource:0}: Error finding container a12a6e9a0e20cf2969e165d985ccfc6dfce9d0b15fc8367011da51bf8eb905a3: Status 404 returned error can't find the container with id a12a6e9a0e20cf2969e165d985ccfc6dfce9d0b15fc8367011da51bf8eb905a3
Sep 10 19:13:26.805728 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.805692    2573 generic.go:356] "Generic (PLEG): container finished" podID="65193eb1-624a-4625-ab5b-eb6068942804" containerID="9e421e1fc218fcb658ac6635cd003ff4f259e66d598082117affa99c1a9b90d6" exitCode=0
Sep 10 19:13:26.806225 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.805783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65193eb1-624a-4625-ab5b-eb6068942804","Type":"ContainerDied","Data":"9e421e1fc218fcb658ac6635cd003ff4f259e66d598082117affa99c1a9b90d6"}
Sep 10 19:13:26.806225 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.805854    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65193eb1-624a-4625-ab5b-eb6068942804","Type":"ContainerStarted","Data":"a12a6e9a0e20cf2969e165d985ccfc6dfce9d0b15fc8367011da51bf8eb905a3"}
Sep 10 19:13:26.835506 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:26.835474    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39748edc-cb63-41e8-b9c0-2c13417157cb" path="/var/lib/kubelet/pods/39748edc-cb63-41e8-b9c0-2c13417157cb/volumes"
Sep 10 19:13:27.820076 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:27.820039    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65193eb1-624a-4625-ab5b-eb6068942804","Type":"ContainerStarted","Data":"bba5cd9f31db41751a2e4a64e8ad78d5827b404a10a87d9c2f02105363c2d949"}
Sep 10 19:13:27.820543 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:27.820142    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65193eb1-624a-4625-ab5b-eb6068942804","Type":"ContainerStarted","Data":"9b26e76cdb7b81176d4ae9d5eb5f75135925f0b7c52e40231367190522c9b510"}
Sep 10 19:13:27.820543 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:27.820216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65193eb1-624a-4625-ab5b-eb6068942804","Type":"ContainerStarted","Data":"c45c940488843e6e493343c011bde4f99e3b052370ac286192ef6f5a436c3b8d"}
Sep 10 19:13:27.820543 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:27.820261    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65193eb1-624a-4625-ab5b-eb6068942804","Type":"ContainerStarted","Data":"0e58ca61b8c9fc2fb94adf22d2149070d8e0e2ea25646a8b2eff3db76d9950b3"}
Sep 10 19:13:27.820543 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:27.820298    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65193eb1-624a-4625-ab5b-eb6068942804","Type":"ContainerStarted","Data":"d0ff90ba6213ca786dd4baacada84f5dfbf0b6db464f74b3f384158c3ef17d42"}
Sep 10 19:13:27.820543 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:27.820338    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"65193eb1-624a-4625-ab5b-eb6068942804","Type":"ContainerStarted","Data":"5fe08c2cc9e4c8f888a9d21968cd78478ced52bcbcb39c31e8a81acb3d74ee5a"}
Sep 10 19:13:27.853121 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:27.853066    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.834037951 podStartE2EDuration="2.85304535s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.007399ms" isStatefulPod=false podCreationTimestamp="2026-09-10 19:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:13:27.851709135 +0000 UTC m=+143.511506339" watchObservedRunningTime="2026-09-10 19:13:27.85304535 +0000 UTC m=+143.512842544"
Sep 10 19:13:31.199400 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:13:31.199365    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:14:26.198788 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:26.198733    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:14:26.215259 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:26.215233    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:14:27.088733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:27.088699    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 19:14:40.048290 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.048251    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xgjms"]
Sep 10 19:14:40.051658 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.051642    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:14:40.054380 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.054356    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 19:14:40.054521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.054387    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 19:14:40.054521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.054400    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7pgjf\""
Sep 10 19:14:40.062106 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.062081    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xgjms"]
Sep 10 19:14:40.136210 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.136143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvt4\" (UniqueName: \"kubernetes.io/projected/481e51e7-e661-4d0c-9bd4-8673bfbfce28-kube-api-access-lqvt4\") pod \"cert-manager-webhook-6486ffd997-xgjms\" (UID: \"481e51e7-e661-4d0c-9bd4-8673bfbfce28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:14:40.136210 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.136217    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481e51e7-e661-4d0c-9bd4-8673bfbfce28-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xgjms\" (UID: \"481e51e7-e661-4d0c-9bd4-8673bfbfce28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:14:40.236835 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.236788    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvt4\" (UniqueName: \"kubernetes.io/projected/481e51e7-e661-4d0c-9bd4-8673bfbfce28-kube-api-access-lqvt4\") pod \"cert-manager-webhook-6486ffd997-xgjms\" (UID: \"481e51e7-e661-4d0c-9bd4-8673bfbfce28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:14:40.236835 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.236841    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481e51e7-e661-4d0c-9bd4-8673bfbfce28-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xgjms\" (UID: \"481e51e7-e661-4d0c-9bd4-8673bfbfce28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:14:40.245615 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.245572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/481e51e7-e661-4d0c-9bd4-8673bfbfce28-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xgjms\" (UID: \"481e51e7-e661-4d0c-9bd4-8673bfbfce28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:14:40.245771 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.245755    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvt4\" (UniqueName: \"kubernetes.io/projected/481e51e7-e661-4d0c-9bd4-8673bfbfce28-kube-api-access-lqvt4\") pod \"cert-manager-webhook-6486ffd997-xgjms\" (UID: \"481e51e7-e661-4d0c-9bd4-8673bfbfce28\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:14:40.373364 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.373276    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:14:40.650114 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:40.650090    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xgjms"]
Sep 10 19:14:40.652622 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:14:40.652595    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod481e51e7_e661_4d0c_9bd4_8673bfbfce28.slice/crio-9a17e4c9516ffdb660f7d12b4b6c79618ef6738d2895edea3620df9662000ab5 WatchSource:0}: Error finding container 9a17e4c9516ffdb660f7d12b4b6c79618ef6738d2895edea3620df9662000ab5: Status 404 returned error can't find the container with id 9a17e4c9516ffdb660f7d12b4b6c79618ef6738d2895edea3620df9662000ab5
Sep 10 19:14:41.137545 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:41.137506    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms" event={"ID":"481e51e7-e661-4d0c-9bd4-8673bfbfce28","Type":"ContainerStarted","Data":"9a17e4c9516ffdb660f7d12b4b6c79618ef6738d2895edea3620df9662000ab5"}
Sep 10 19:14:44.155577 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:44.155544    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms" event={"ID":"481e51e7-e661-4d0c-9bd4-8673bfbfce28","Type":"ContainerStarted","Data":"bfbe9cfaa47924ae5dc9a285e234427684bf17dbfead2fcf66985c8fbeedec76"}
Sep 10 19:14:44.155967 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:44.155663    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:14:44.184038 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:44.183988    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms" podStartSLOduration=0.811890204 podStartE2EDuration="4.183972077s" totalImagesPullingTime="3.372081873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:14:44.183390699 +0000 UTC m=+219.843187916" watchObservedRunningTime="2026-09-10 19:14:44.183972077 +0000 UTC m=+219.843769270"
Sep 10 19:14:45.709613 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.709581    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qm5jl"]
Sep 10 19:14:45.713244 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.713228    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qm5jl"
Sep 10 19:14:45.716480 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.716454    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7h5qq\""
Sep 10 19:14:45.729126 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.729101    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qm5jl"]
Sep 10 19:14:45.791616 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.791581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43c5d9fe-6a34-4abb-86f5-907962efeab8-bound-sa-token\") pod \"cert-manager-55b4d954d5-qm5jl\" (UID: \"43c5d9fe-6a34-4abb-86f5-907962efeab8\") " pod="cert-manager/cert-manager-55b4d954d5-qm5jl"
Sep 10 19:14:45.791780 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.791635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvm8q\" (UniqueName: \"kubernetes.io/projected/43c5d9fe-6a34-4abb-86f5-907962efeab8-kube-api-access-lvm8q\") pod \"cert-manager-55b4d954d5-qm5jl\" (UID: \"43c5d9fe-6a34-4abb-86f5-907962efeab8\") " pod="cert-manager/cert-manager-55b4d954d5-qm5jl"
Sep 10 19:14:45.892773 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.892731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvm8q\" (UniqueName: \"kubernetes.io/projected/43c5d9fe-6a34-4abb-86f5-907962efeab8-kube-api-access-lvm8q\") pod \"cert-manager-55b4d954d5-qm5jl\" (UID: \"43c5d9fe-6a34-4abb-86f5-907962efeab8\") " pod="cert-manager/cert-manager-55b4d954d5-qm5jl"
Sep 10 19:14:45.892970 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.892852    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43c5d9fe-6a34-4abb-86f5-907962efeab8-bound-sa-token\") pod \"cert-manager-55b4d954d5-qm5jl\" (UID: \"43c5d9fe-6a34-4abb-86f5-907962efeab8\") " pod="cert-manager/cert-manager-55b4d954d5-qm5jl"
Sep 10 19:14:45.907920 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.907890    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/43c5d9fe-6a34-4abb-86f5-907962efeab8-bound-sa-token\") pod \"cert-manager-55b4d954d5-qm5jl\" (UID: \"43c5d9fe-6a34-4abb-86f5-907962efeab8\") " pod="cert-manager/cert-manager-55b4d954d5-qm5jl"
Sep 10 19:14:45.908056 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:45.907895    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvm8q\" (UniqueName: \"kubernetes.io/projected/43c5d9fe-6a34-4abb-86f5-907962efeab8-kube-api-access-lvm8q\") pod \"cert-manager-55b4d954d5-qm5jl\" (UID: \"43c5d9fe-6a34-4abb-86f5-907962efeab8\") " pod="cert-manager/cert-manager-55b4d954d5-qm5jl"
Sep 10 19:14:46.021735 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:46.021710    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qm5jl"
Sep 10 19:14:46.306106 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:46.306046    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qm5jl"]
Sep 10 19:14:46.308953 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:14:46.308922    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43c5d9fe_6a34_4abb_86f5_907962efeab8.slice/crio-bbe7258824f7b7d189486c7dd5a7cb76fbf9280b6e0557eefa52c6710280ca00 WatchSource:0}: Error finding container bbe7258824f7b7d189486c7dd5a7cb76fbf9280b6e0557eefa52c6710280ca00: Status 404 returned error can't find the container with id bbe7258824f7b7d189486c7dd5a7cb76fbf9280b6e0557eefa52c6710280ca00
Sep 10 19:14:47.169456 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:47.169421    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qm5jl" event={"ID":"43c5d9fe-6a34-4abb-86f5-907962efeab8","Type":"ContainerStarted","Data":"304f6e748ace2fec193ba719d1ca25c3aa32be764924e6883b8c3ff15669b2dc"}
Sep 10 19:14:47.169456 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:47.169464    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qm5jl" event={"ID":"43c5d9fe-6a34-4abb-86f5-907962efeab8","Type":"ContainerStarted","Data":"bbe7258824f7b7d189486c7dd5a7cb76fbf9280b6e0557eefa52c6710280ca00"}
Sep 10 19:14:47.207496 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:47.207442    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-qm5jl" podStartSLOduration=2.207422782 podStartE2EDuration="2.207422782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:14:47.203743753 +0000 UTC m=+222.863540945" watchObservedRunningTime="2026-09-10 19:14:47.207422782 +0000 UTC m=+222.867219974"
Sep 10 19:14:50.161753 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:14:50.161724    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xgjms"
Sep 10 19:15:07.502811 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.502767    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"]
Sep 10 19:15:07.506410 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.506385    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.510949 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.510926    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:15:07.510949 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.510931    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 19:15:07.512916 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.512894    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 19:15:07.513045 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.512966    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 19:15:07.517365 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.517347    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jlvx4\""
Sep 10 19:15:07.518635 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.518620    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 19:15:07.524056 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.524038    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"]
Sep 10 19:15:07.589813 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.589775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-cert\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.589813 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.589812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-metrics-cert\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.590026 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.589840    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-manager-config\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.590026 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.589949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq8qd\" (UniqueName: \"kubernetes.io/projected/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-kube-api-access-mq8qd\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.691316 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.691258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-cert\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.691316 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.691328    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-metrics-cert\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.691563 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.691392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-manager-config\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.691563 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.691461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq8qd\" (UniqueName: \"kubernetes.io/projected/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-kube-api-access-mq8qd\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.692089 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.692065    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-manager-config\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.693951 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.693930    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-metrics-cert\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.694190 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.694144    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-cert\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.702938 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.702915    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq8qd\" (UniqueName: \"kubernetes.io/projected/6d33755e-8af6-4435-bba7-6b0f9cb6f17e-kube-api-access-mq8qd\") pod \"lws-controller-manager-6748c6bd4b-rmpm8\" (UID: \"6d33755e-8af6-4435-bba7-6b0f9cb6f17e\") " pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:07.815877 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:07.815843    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:08.104496 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:08.104473    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"]
Sep 10 19:15:08.106754 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:15:08.106724    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d33755e_8af6_4435_bba7_6b0f9cb6f17e.slice/crio-0e1dfd97901bac99b446e50df89921da4f0da8c0617feb763474873853262718 WatchSource:0}: Error finding container 0e1dfd97901bac99b446e50df89921da4f0da8c0617feb763474873853262718: Status 404 returned error can't find the container with id 0e1dfd97901bac99b446e50df89921da4f0da8c0617feb763474873853262718
Sep 10 19:15:08.257088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:08.257053    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8" event={"ID":"6d33755e-8af6-4435-bba7-6b0f9cb6f17e","Type":"ContainerStarted","Data":"0e1dfd97901bac99b446e50df89921da4f0da8c0617feb763474873853262718"}
Sep 10 19:15:10.333138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.333107    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"]
Sep 10 19:15:10.338462 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.338441    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.342309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.342285    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 19:15:10.342309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.342314    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 19:15:10.342575 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.342555    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 19:15:10.342707 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.342646    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-574n2\""
Sep 10 19:15:10.342707 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.342678    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 19:15:10.363249 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.363218    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"]
Sep 10 19:15:10.414202 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.414169    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11bd4506-ecf3-4b70-b37f-d39b75348869-manifests\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.414398 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.414231    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11bd4506-ecf3-4b70-b37f-d39b75348869-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.414398 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.414299    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11bd4506-ecf3-4b70-b37f-d39b75348869-tmp\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.414398 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.414351    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11bd4506-ecf3-4b70-b37f-d39b75348869-webhook-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.414548 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.414439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz49c\" (UniqueName: \"kubernetes.io/projected/11bd4506-ecf3-4b70-b37f-d39b75348869-kube-api-access-pz49c\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.514904 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.514866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11bd4506-ecf3-4b70-b37f-d39b75348869-webhook-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.515106 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.514916    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz49c\" (UniqueName: \"kubernetes.io/projected/11bd4506-ecf3-4b70-b37f-d39b75348869-kube-api-access-pz49c\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.515106 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.514954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11bd4506-ecf3-4b70-b37f-d39b75348869-manifests\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.515106 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.514995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11bd4506-ecf3-4b70-b37f-d39b75348869-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.515447 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.515421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11bd4506-ecf3-4b70-b37f-d39b75348869-tmp\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.515557 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.515487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11bd4506-ecf3-4b70-b37f-d39b75348869-manifests\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.517838 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.517811    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11bd4506-ecf3-4b70-b37f-d39b75348869-webhook-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.518059 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.518042    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11bd4506-ecf3-4b70-b37f-d39b75348869-tmp\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.518366 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.518345    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11bd4506-ecf3-4b70-b37f-d39b75348869-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.525240 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.525213    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz49c\" (UniqueName: \"kubernetes.io/projected/11bd4506-ecf3-4b70-b37f-d39b75348869-kube-api-access-pz49c\") pod \"opendatahub-operator-controller-manager-d54bdf466-xsg8v\" (UID: \"11bd4506-ecf3-4b70-b37f-d39b75348869\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:10.651486 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:10.651395    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:11.156623 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:11.156597    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"]
Sep 10 19:15:11.158705 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:15:11.158678    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bd4506_ecf3_4b70_b37f_d39b75348869.slice/crio-41d48d720f468e2d8aaaf02b989b416760fe0dfe183b93aa009fbd4dfda22115 WatchSource:0}: Error finding container 41d48d720f468e2d8aaaf02b989b416760fe0dfe183b93aa009fbd4dfda22115: Status 404 returned error can't find the container with id 41d48d720f468e2d8aaaf02b989b416760fe0dfe183b93aa009fbd4dfda22115
Sep 10 19:15:11.273353 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:11.273301    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v" event={"ID":"11bd4506-ecf3-4b70-b37f-d39b75348869","Type":"ContainerStarted","Data":"41d48d720f468e2d8aaaf02b989b416760fe0dfe183b93aa009fbd4dfda22115"}
Sep 10 19:15:11.274780 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:11.274756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8" event={"ID":"6d33755e-8af6-4435-bba7-6b0f9cb6f17e","Type":"ContainerStarted","Data":"52a5581df5dc563158763bc22aa17863828a4efe414419f2db57bc8521b3116e"}
Sep 10 19:15:11.274912 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:11.274887    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:15.300535 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:15.300494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v" event={"ID":"11bd4506-ecf3-4b70-b37f-d39b75348869","Type":"ContainerStarted","Data":"0abb4665c7af8690e462936d129cf723e4540cb8987fcf9dcecc120f7a3876f2"}
Sep 10 19:15:15.337521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:15.337463    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8" podStartSLOduration=5.504540419 podStartE2EDuration="8.337444441s" totalImagesPullingTime="2.832904022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:11.303818045 +0000 UTC m=+246.963615234" watchObservedRunningTime="2026-09-10 19:15:15.337444441 +0000 UTC m=+250.997241702"
Sep 10 19:15:16.306897 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:16.306860    2573 generic.go:356] "Generic (PLEG): container finished" podID="11bd4506-ecf3-4b70-b37f-d39b75348869" containerID="0abb4665c7af8690e462936d129cf723e4540cb8987fcf9dcecc120f7a3876f2" exitCode=0
Sep 10 19:15:16.307312 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:16.306957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v" event={"ID":"11bd4506-ecf3-4b70-b37f-d39b75348869","Type":"ContainerDied","Data":"0abb4665c7af8690e462936d129cf723e4540cb8987fcf9dcecc120f7a3876f2"}
Sep 10 19:15:17.314334 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:17.314299    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v" event={"ID":"11bd4506-ecf3-4b70-b37f-d39b75348869","Type":"ContainerStarted","Data":"97f5484b86f44e0c749257d2517100027b15521e4c3eb10648f309c3f6e1422f"}
Sep 10 19:15:17.314713 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:17.314456    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:17.349876 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:17.349824    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v" podStartSLOduration=2.723869518 podStartE2EDuration="7.349807146s" totalImagesPullingTime="3.476699129s" totalInitContainerRuntime="1.149238499s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:17.347124685 +0000 UTC m=+253.006921878" watchObservedRunningTime="2026-09-10 19:15:17.349807146 +0000 UTC m=+253.009604393"
Sep 10 19:15:22.284782 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:22.284745    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6748c6bd4b-rmpm8"
Sep 10 19:15:28.321244 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:28.321138    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-xsg8v"
Sep 10 19:15:30.666767 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.666731    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"]
Sep 10 19:15:30.670407 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.670380    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.673894 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.673864    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 19:15:30.674041 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.673903    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 19:15:30.674041 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.673865    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 19:15:30.683470 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.683448    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 19:15:30.683470 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.683463    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h96l2\""
Sep 10 19:15:30.703078 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.703045    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"]
Sep 10 19:15:30.797435 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.797401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075a6585-5c1b-4f46-911d-2f7b26c45cb2-tls-certs\") pod \"kube-auth-proxy-6b5f7cd94b-cqhc8\" (UID: \"075a6585-5c1b-4f46-911d-2f7b26c45cb2\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.797612 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.797458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl558\" (UniqueName: \"kubernetes.io/projected/075a6585-5c1b-4f46-911d-2f7b26c45cb2-kube-api-access-zl558\") pod \"kube-auth-proxy-6b5f7cd94b-cqhc8\" (UID: \"075a6585-5c1b-4f46-911d-2f7b26c45cb2\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.797612 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.797543    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/075a6585-5c1b-4f46-911d-2f7b26c45cb2-tmp\") pod \"kube-auth-proxy-6b5f7cd94b-cqhc8\" (UID: \"075a6585-5c1b-4f46-911d-2f7b26c45cb2\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.898242 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.898182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/075a6585-5c1b-4f46-911d-2f7b26c45cb2-tmp\") pod \"kube-auth-proxy-6b5f7cd94b-cqhc8\" (UID: \"075a6585-5c1b-4f46-911d-2f7b26c45cb2\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.898433 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.898283    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075a6585-5c1b-4f46-911d-2f7b26c45cb2-tls-certs\") pod \"kube-auth-proxy-6b5f7cd94b-cqhc8\" (UID: \"075a6585-5c1b-4f46-911d-2f7b26c45cb2\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.898433 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.898331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl558\" (UniqueName: \"kubernetes.io/projected/075a6585-5c1b-4f46-911d-2f7b26c45cb2-kube-api-access-zl558\") pod \"kube-auth-proxy-6b5f7cd94b-cqhc8\" (UID: \"075a6585-5c1b-4f46-911d-2f7b26c45cb2\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.900600 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.900574    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/075a6585-5c1b-4f46-911d-2f7b26c45cb2-tmp\") pod \"kube-auth-proxy-6b5f7cd94b-cqhc8\" (UID: \"075a6585-5c1b-4f46-911d-2f7b26c45cb2\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.900873 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.900854    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075a6585-5c1b-4f46-911d-2f7b26c45cb2-tls-certs\") pod \"kube-auth-proxy-6b5f7cd94b-cqhc8\" (UID: \"075a6585-5c1b-4f46-911d-2f7b26c45cb2\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.943476 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.943406    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl558\" (UniqueName: \"kubernetes.io/projected/075a6585-5c1b-4f46-911d-2f7b26c45cb2-kube-api-access-zl558\") pod \"kube-auth-proxy-6b5f7cd94b-cqhc8\" (UID: \"075a6585-5c1b-4f46-911d-2f7b26c45cb2\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:30.979962 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:30.979927    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"
Sep 10 19:15:31.285890 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:15:31.285860    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075a6585_5c1b_4f46_911d_2f7b26c45cb2.slice/crio-da13c18fa4883169e45d5b70c7bad8d401f8481cf4bb2d701de8e2e7e7863a75 WatchSource:0}: Error finding container da13c18fa4883169e45d5b70c7bad8d401f8481cf4bb2d701de8e2e7e7863a75: Status 404 returned error can't find the container with id da13c18fa4883169e45d5b70c7bad8d401f8481cf4bb2d701de8e2e7e7863a75
Sep 10 19:15:31.286317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:31.286295    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8"]
Sep 10 19:15:31.377782 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:31.377743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8" event={"ID":"075a6585-5c1b-4f46-911d-2f7b26c45cb2","Type":"ContainerStarted","Data":"da13c18fa4883169e45d5b70c7bad8d401f8481cf4bb2d701de8e2e7e7863a75"}
Sep 10 19:15:35.400478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:35.400436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8" event={"ID":"075a6585-5c1b-4f46-911d-2f7b26c45cb2","Type":"ContainerStarted","Data":"75d2133c2de912fac088691ce4348f280d95f6a979fe14b694da21de18865275"}
Sep 10 19:15:35.424893 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:15:35.424830    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-cqhc8" podStartSLOduration=1.8796983360000001 podStartE2EDuration="5.424813064s" totalImagesPullingTime="3.545114728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:35.42236136 +0000 UTC m=+271.082158553" watchObservedRunningTime="2026-09-10 19:15:35.424813064 +0000 UTC m=+271.084610263"
Sep 10 19:16:04.766828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:16:04.766767    2573 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 19:17:06.179015 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.178977    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54546fdd-2xtb9"]
Sep 10 19:17:06.182414 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.182386    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.189017 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.188996    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 19:17:06.189176 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.189077    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 19:17:06.190305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.189709    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 19:17:06.190305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.189749    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 19:17:06.190305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.190062    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 19:17:06.193339 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.193310    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fnqjb\""
Sep 10 19:17:06.199978 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.199957    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 19:17:06.219744 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.219716    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54546fdd-2xtb9"]
Sep 10 19:17:06.251465 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.251407    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-oauth-serving-cert\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.251638 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.251478    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52ffe80b-6e8f-4714-a2e4-16827b343745-console-serving-cert\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.251638 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.251504    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52ffe80b-6e8f-4714-a2e4-16827b343745-console-oauth-config\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.251638 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.251533    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-service-ca\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.251763 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.251656    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn4g5\" (UniqueName: \"kubernetes.io/projected/52ffe80b-6e8f-4714-a2e4-16827b343745-kube-api-access-pn4g5\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.251763 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.251729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-console-config\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.251849 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.251785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-trusted-ca-bundle\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.353100 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.353056    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn4g5\" (UniqueName: \"kubernetes.io/projected/52ffe80b-6e8f-4714-a2e4-16827b343745-kube-api-access-pn4g5\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.353310 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.353126    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-console-config\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.353310 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.353195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-trusted-ca-bundle\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.353310 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.353256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-oauth-serving-cert\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.353310 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.353294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52ffe80b-6e8f-4714-a2e4-16827b343745-console-serving-cert\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.353542 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.353324    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52ffe80b-6e8f-4714-a2e4-16827b343745-console-oauth-config\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.353542 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.353360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-service-ca\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.354267 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.354238    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-service-ca\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.354391 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.354305    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-oauth-serving-cert\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.354391 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.354303    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-trusted-ca-bundle\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.354708 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.354691    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52ffe80b-6e8f-4714-a2e4-16827b343745-console-config\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.356575 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.356551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52ffe80b-6e8f-4714-a2e4-16827b343745-console-serving-cert\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.356696 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.356663    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52ffe80b-6e8f-4714-a2e4-16827b343745-console-oauth-config\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.387386 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.387357    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn4g5\" (UniqueName: \"kubernetes.io/projected/52ffe80b-6e8f-4714-a2e4-16827b343745-kube-api-access-pn4g5\") pod \"console-54546fdd-2xtb9\" (UID: \"52ffe80b-6e8f-4714-a2e4-16827b343745\") " pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.493384 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.493288    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:06.774372 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.774344    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54546fdd-2xtb9"]
Sep 10 19:17:06.776947 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:17:06.776919    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ffe80b_6e8f_4714_a2e4_16827b343745.slice/crio-7e7517ce2b71869524609aa6da136f954a941ed89573f3a9db5dad3d70904b2b WatchSource:0}: Error finding container 7e7517ce2b71869524609aa6da136f954a941ed89573f3a9db5dad3d70904b2b: Status 404 returned error can't find the container with id 7e7517ce2b71869524609aa6da136f954a941ed89573f3a9db5dad3d70904b2b
Sep 10 19:17:06.822189 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:06.822132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54546fdd-2xtb9" event={"ID":"52ffe80b-6e8f-4714-a2e4-16827b343745","Type":"ContainerStarted","Data":"7e7517ce2b71869524609aa6da136f954a941ed89573f3a9db5dad3d70904b2b"}
Sep 10 19:17:07.828453 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:07.828424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54546fdd-2xtb9" event={"ID":"52ffe80b-6e8f-4714-a2e4-16827b343745","Type":"ContainerStarted","Data":"6ca7b267805d23ed7e59caff3e458050acccaf8feb76d92418f92b9d7ff5e376"}
Sep 10 19:17:16.493788 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:16.493733    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:16.494233 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:16.493918    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:16.499046 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:16.499018    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:16.519811 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:16.519756    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54546fdd-2xtb9" podStartSLOduration=10.519735409 podStartE2EDuration="10.519735409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:07.884829327 +0000 UTC m=+363.544626519" watchObservedRunningTime="2026-09-10 19:17:16.519735409 +0000 UTC m=+372.179532672"
Sep 10 19:17:16.871396 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:16.871369    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54546fdd-2xtb9"
Sep 10 19:17:20.362504 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.362465    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"]
Sep 10 19:17:20.366427 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.366400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"
Sep 10 19:17:20.369358 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.369336    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-j8r55\""
Sep 10 19:17:20.370462 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.370443    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 19:17:20.370572 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.370447    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 19:17:20.382513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.382480    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"]
Sep 10 19:17:20.390944 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.390909    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnvqn\" (UniqueName: \"kubernetes.io/projected/17998649-d79e-4c82-91c3-a005e54c004c-kube-api-access-mnvqn\") pod \"limitador-operator-controller-manager-74495c69f8-stz98\" (UID: \"17998649-d79e-4c82-91c3-a005e54c004c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"
Sep 10 19:17:20.492022 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.491981    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnvqn\" (UniqueName: \"kubernetes.io/projected/17998649-d79e-4c82-91c3-a005e54c004c-kube-api-access-mnvqn\") pod \"limitador-operator-controller-manager-74495c69f8-stz98\" (UID: \"17998649-d79e-4c82-91c3-a005e54c004c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"
Sep 10 19:17:20.511976 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.511938    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnvqn\" (UniqueName: \"kubernetes.io/projected/17998649-d79e-4c82-91c3-a005e54c004c-kube-api-access-mnvqn\") pod \"limitador-operator-controller-manager-74495c69f8-stz98\" (UID: \"17998649-d79e-4c82-91c3-a005e54c004c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"
Sep 10 19:17:20.677876 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.677792    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"
Sep 10 19:17:20.962393 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.962310    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"]
Sep 10 19:17:20.965742 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:17:20.965713    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17998649_d79e_4c82_91c3_a005e54c004c.slice/crio-5ea6fbd797bdb18833840268c86b78860d91687aa81b7930ceb301d4b7a4f7f4 WatchSource:0}: Error finding container 5ea6fbd797bdb18833840268c86b78860d91687aa81b7930ceb301d4b7a4f7f4: Status 404 returned error can't find the container with id 5ea6fbd797bdb18833840268c86b78860d91687aa81b7930ceb301d4b7a4f7f4
Sep 10 19:17:20.968198 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:20.968179    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:17:21.892989 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:21.892951    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98" event={"ID":"17998649-d79e-4c82-91c3-a005e54c004c","Type":"ContainerStarted","Data":"5ea6fbd797bdb18833840268c86b78860d91687aa81b7930ceb301d4b7a4f7f4"}
Sep 10 19:17:23.905697 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:23.905662    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98" event={"ID":"17998649-d79e-4c82-91c3-a005e54c004c","Type":"ContainerStarted","Data":"bc50069cbc60282d6ea1c2ba8cbc88096dc601e103b6df242484c65b045ff83a"}
Sep 10 19:17:23.906086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:23.905749    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"
Sep 10 19:17:23.927982 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:23.927887    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98" podStartSLOduration=1.2172147309999999 podStartE2EDuration="3.927872087s" totalImagesPullingTime="2.710657356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:23.927092139 +0000 UTC m=+379.586889333" watchObservedRunningTime="2026-09-10 19:17:23.927872087 +0000 UTC m=+379.587669279"
Sep 10 19:17:34.913809 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:34.913780    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-stz98"
Sep 10 19:17:54.874166 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:54.874113    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9rrw5"]
Sep 10 19:17:54.882825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:54.882799    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5"
Sep 10 19:17:54.886108 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:54.886087    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-77n47\""
Sep 10 19:17:54.891028 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:54.891003    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9rrw5"]
Sep 10 19:17:54.911545 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:54.911507    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb6hz\" (UniqueName: \"kubernetes.io/projected/89a9a62f-600b-46e9-b8c3-b481b6d09d6b-kube-api-access-fb6hz\") pod \"authorino-7bcfd8f9cd-9rrw5\" (UID: \"89a9a62f-600b-46e9-b8c3-b481b6d09d6b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5"
Sep 10 19:17:55.012012 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.011976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb6hz\" (UniqueName: \"kubernetes.io/projected/89a9a62f-600b-46e9-b8c3-b481b6d09d6b-kube-api-access-fb6hz\") pod \"authorino-7bcfd8f9cd-9rrw5\" (UID: \"89a9a62f-600b-46e9-b8c3-b481b6d09d6b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5"
Sep 10 19:17:55.027634 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.027600    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-g2hn2"]
Sep 10 19:17:55.031127 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.031099    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb6hz\" (UniqueName: \"kubernetes.io/projected/89a9a62f-600b-46e9-b8c3-b481b6d09d6b-kube-api-access-fb6hz\") pod \"authorino-7bcfd8f9cd-9rrw5\" (UID: \"89a9a62f-600b-46e9-b8c3-b481b6d09d6b\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5"
Sep 10 19:17:55.031433 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.031418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-g2hn2"
Sep 10 19:17:55.039479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.039453    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-g2hn2"]
Sep 10 19:17:55.112760 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.112721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvc6n\" (UniqueName: \"kubernetes.io/projected/0183005d-b99d-49c4-8b35-fd8e2040c817-kube-api-access-jvc6n\") pod \"authorino-59c974bd48-g2hn2\" (UID: \"0183005d-b99d-49c4-8b35-fd8e2040c817\") " pod="kuadrant-system/authorino-59c974bd48-g2hn2"
Sep 10 19:17:55.192641 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.192561    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5"
Sep 10 19:17:55.213321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.213293    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvc6n\" (UniqueName: \"kubernetes.io/projected/0183005d-b99d-49c4-8b35-fd8e2040c817-kube-api-access-jvc6n\") pod \"authorino-59c974bd48-g2hn2\" (UID: \"0183005d-b99d-49c4-8b35-fd8e2040c817\") " pod="kuadrant-system/authorino-59c974bd48-g2hn2"
Sep 10 19:17:55.222210 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.222176    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvc6n\" (UniqueName: \"kubernetes.io/projected/0183005d-b99d-49c4-8b35-fd8e2040c817-kube-api-access-jvc6n\") pod \"authorino-59c974bd48-g2hn2\" (UID: \"0183005d-b99d-49c4-8b35-fd8e2040c817\") " pod="kuadrant-system/authorino-59c974bd48-g2hn2"
Sep 10 19:17:55.348483 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.348435    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-g2hn2"
Sep 10 19:17:55.463122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.463099    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9rrw5"]
Sep 10 19:17:55.465349 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:17:55.465322    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89a9a62f_600b_46e9_b8c3_b481b6d09d6b.slice/crio-1344af60595d83c9b81da0e982426a9d28493c1db5dea300cfe0be1d40581147 WatchSource:0}: Error finding container 1344af60595d83c9b81da0e982426a9d28493c1db5dea300cfe0be1d40581147: Status 404 returned error can't find the container with id 1344af60595d83c9b81da0e982426a9d28493c1db5dea300cfe0be1d40581147
Sep 10 19:17:55.624118 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:55.624086    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-g2hn2"]
Sep 10 19:17:55.626291 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:17:55.626268    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0183005d_b99d_49c4_8b35_fd8e2040c817.slice/crio-3ce0eb2a907309905286f6fa8814e79feda51fa33568d3a039bf6d762acd792f WatchSource:0}: Error finding container 3ce0eb2a907309905286f6fa8814e79feda51fa33568d3a039bf6d762acd792f: Status 404 returned error can't find the container with id 3ce0eb2a907309905286f6fa8814e79feda51fa33568d3a039bf6d762acd792f
Sep 10 19:17:56.056285 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:56.056248    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-g2hn2" event={"ID":"0183005d-b99d-49c4-8b35-fd8e2040c817","Type":"ContainerStarted","Data":"3ce0eb2a907309905286f6fa8814e79feda51fa33568d3a039bf6d762acd792f"}
Sep 10 19:17:56.057386 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:56.057363    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5" event={"ID":"89a9a62f-600b-46e9-b8c3-b481b6d09d6b","Type":"ContainerStarted","Data":"1344af60595d83c9b81da0e982426a9d28493c1db5dea300cfe0be1d40581147"}
Sep 10 19:17:58.069462 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:58.069422    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5" event={"ID":"89a9a62f-600b-46e9-b8c3-b481b6d09d6b","Type":"ContainerStarted","Data":"1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d"}
Sep 10 19:17:58.070902 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:58.070878    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-g2hn2" event={"ID":"0183005d-b99d-49c4-8b35-fd8e2040c817","Type":"ContainerStarted","Data":"5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6"}
Sep 10 19:17:58.087267 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:58.087213    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5" podStartSLOduration=2.147564981 podStartE2EDuration="4.087189926s" totalImagesPullingTime="1.939624945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:58.085236877 +0000 UTC m=+413.745034092" watchObservedRunningTime="2026-09-10 19:17:58.087189926 +0000 UTC m=+413.746987124"
Sep 10 19:17:58.100808 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:58.100746    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-g2hn2" podStartSLOduration=1.325929192 podStartE2EDuration="3.100724707s" totalImagesPullingTime="1.774795515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:58.100682096 +0000 UTC m=+413.760479290" watchObservedRunningTime="2026-09-10 19:17:58.100724707 +0000 UTC m=+413.760521900"
Sep 10 19:17:58.136105 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:17:58.136066    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9rrw5"]
Sep 10 19:18:00.080681 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:00.080639    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5" podUID="89a9a62f-600b-46e9-b8c3-b481b6d09d6b" containerName="authorino" containerID="cri-o://1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d" gracePeriod=30
Sep 10 19:18:00.328804 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:00.328777    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5"
Sep 10 19:18:00.362834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:00.362764    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb6hz\" (UniqueName: \"kubernetes.io/projected/89a9a62f-600b-46e9-b8c3-b481b6d09d6b-kube-api-access-fb6hz\") pod \"89a9a62f-600b-46e9-b8c3-b481b6d09d6b\" (UID: \"89a9a62f-600b-46e9-b8c3-b481b6d09d6b\") "
Sep 10 19:18:00.365113 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:00.365092    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a9a62f-600b-46e9-b8c3-b481b6d09d6b-kube-api-access-fb6hz" (OuterVolumeSpecName: "kube-api-access-fb6hz") pod "89a9a62f-600b-46e9-b8c3-b481b6d09d6b" (UID: "89a9a62f-600b-46e9-b8c3-b481b6d09d6b"). InnerVolumeSpecName "kube-api-access-fb6hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:18:00.464090 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:00.464052    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb6hz\" (UniqueName: \"kubernetes.io/projected/89a9a62f-600b-46e9-b8c3-b481b6d09d6b-kube-api-access-fb6hz\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:18:01.085976 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:01.085930    2573 generic.go:356] "Generic (PLEG): container finished" podID="89a9a62f-600b-46e9-b8c3-b481b6d09d6b" containerID="1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d" exitCode=0
Sep 10 19:18:01.085976 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:01.085977    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5"
Sep 10 19:18:01.086440 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:01.085995    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5" event={"ID":"89a9a62f-600b-46e9-b8c3-b481b6d09d6b","Type":"ContainerDied","Data":"1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d"}
Sep 10 19:18:01.086440 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:01.086032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9rrw5" event={"ID":"89a9a62f-600b-46e9-b8c3-b481b6d09d6b","Type":"ContainerDied","Data":"1344af60595d83c9b81da0e982426a9d28493c1db5dea300cfe0be1d40581147"}
Sep 10 19:18:01.086440 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:01.086050    2573 scope.go:117] "RemoveContainer" containerID="1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d"
Sep 10 19:18:01.094820 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:01.094799    2573 scope.go:117] "RemoveContainer" containerID="1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d"
Sep 10 19:18:01.095105 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:18:01.095088    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d\": container with ID starting with 1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d not found: ID does not exist" containerID="1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d"
Sep 10 19:18:01.095184 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:01.095114    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d"} err="failed to get container status \"1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d\": rpc error: code = NotFound desc = could not find container \"1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d\": container with ID starting with 1985a5faca937cd39a9a7cb754dddb2fe66058031aa339e024181abb3097580d not found: ID does not exist"
Sep 10 19:18:01.101527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:01.101507    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9rrw5"]
Sep 10 19:18:01.105292 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:01.105273    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9rrw5"]
Sep 10 19:18:02.834080 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:02.834047    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89a9a62f-600b-46e9-b8c3-b481b6d09d6b" path="/var/lib/kubelet/pods/89a9a62f-600b-46e9-b8c3-b481b6d09d6b/volumes"
Sep 10 19:18:10.641738 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.641699    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"]
Sep 10 19:18:10.642501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.642167    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a9a62f-600b-46e9-b8c3-b481b6d09d6b" containerName="authorino"
Sep 10 19:18:10.642501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.642179    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a9a62f-600b-46e9-b8c3-b481b6d09d6b" containerName="authorino"
Sep 10 19:18:10.642501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.642253    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a9a62f-600b-46e9-b8c3-b481b6d09d6b" containerName="authorino"
Sep 10 19:18:10.646644 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.646620    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:10.649966 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.649934    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-z2jdz\""
Sep 10 19:18:10.650361 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.650345    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 19:18:10.651255 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.651226    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 19:18:10.651359 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.651264    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 19:18:10.668276 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.668251    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"]
Sep 10 19:18:10.756295 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.756259    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3171ecb9-0f9b-4a39-88de-12953b2ae50e-data\") pod \"postgres-69fb4594cf-rczrc\" (UID: \"3171ecb9-0f9b-4a39-88de-12953b2ae50e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:10.756467 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.756304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4f29\" (UniqueName: \"kubernetes.io/projected/3171ecb9-0f9b-4a39-88de-12953b2ae50e-kube-api-access-l4f29\") pod \"postgres-69fb4594cf-rczrc\" (UID: \"3171ecb9-0f9b-4a39-88de-12953b2ae50e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:10.857389 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.857350    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3171ecb9-0f9b-4a39-88de-12953b2ae50e-data\") pod \"postgres-69fb4594cf-rczrc\" (UID: \"3171ecb9-0f9b-4a39-88de-12953b2ae50e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:10.857389 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.857396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4f29\" (UniqueName: \"kubernetes.io/projected/3171ecb9-0f9b-4a39-88de-12953b2ae50e-kube-api-access-l4f29\") pod \"postgres-69fb4594cf-rczrc\" (UID: \"3171ecb9-0f9b-4a39-88de-12953b2ae50e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:10.857731 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.857710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3171ecb9-0f9b-4a39-88de-12953b2ae50e-data\") pod \"postgres-69fb4594cf-rczrc\" (UID: \"3171ecb9-0f9b-4a39-88de-12953b2ae50e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:10.868850 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.868819    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4f29\" (UniqueName: \"kubernetes.io/projected/3171ecb9-0f9b-4a39-88de-12953b2ae50e-kube-api-access-l4f29\") pod \"postgres-69fb4594cf-rczrc\" (UID: \"3171ecb9-0f9b-4a39-88de-12953b2ae50e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:10.957992 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:10.957913    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:11.237834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:11.237807    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"]
Sep 10 19:18:11.240391 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:18:11.240361    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3171ecb9_0f9b_4a39_88de_12953b2ae50e.slice/crio-60aed67b3a72891d365e5f24c4acd26160d7d917729ce6e10be317b1e296dedc WatchSource:0}: Error finding container 60aed67b3a72891d365e5f24c4acd26160d7d917729ce6e10be317b1e296dedc: Status 404 returned error can't find the container with id 60aed67b3a72891d365e5f24c4acd26160d7d917729ce6e10be317b1e296dedc
Sep 10 19:18:12.143726 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:12.143677    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc" event={"ID":"3171ecb9-0f9b-4a39-88de-12953b2ae50e","Type":"ContainerStarted","Data":"60aed67b3a72891d365e5f24c4acd26160d7d917729ce6e10be317b1e296dedc"}
Sep 10 19:18:17.173969 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:17.173935    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc" event={"ID":"3171ecb9-0f9b-4a39-88de-12953b2ae50e","Type":"ContainerStarted","Data":"5f8f687f645f978ef916f2b3b9bfa64eacf0f1736acc2b9e3ab3886cb288f6f9"}
Sep 10 19:18:17.174378 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:17.174044    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:17.195439 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:17.195387    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc" podStartSLOduration=2.133998517 podStartE2EDuration="7.195370738s" totalImagesPullingTime="5.061372221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:18:17.192702453 +0000 UTC m=+432.852499645" watchObservedRunningTime="2026-09-10 19:18:17.195370738 +0000 UTC m=+432.855167934"
Sep 10 19:18:23.208875 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.208847    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rczrc"
Sep 10 19:18:23.723124 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.723095    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-nw47h"]
Sep 10 19:18:23.730120 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.730100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nw47h"
Sep 10 19:18:23.733877 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.733850    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-nw47h"]
Sep 10 19:18:23.796684 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.796646    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nw47h"]
Sep 10 19:18:23.796938 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:18:23.796917    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4hlv8], unattached volumes=[], failed to process volumes=[kube-api-access-4hlv8]: context canceled" pod="kuadrant-system/authorino-557f876f69-nw47h" podUID="9ae42a94-d32c-4653-ba9e-6701dc26423f"
Sep 10 19:18:23.829914 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.829882    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bpcq2"]
Sep 10 19:18:23.833643 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.833625    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:23.836307 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.836290    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 19:18:23.842178 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.842141    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bpcq2"]
Sep 10 19:18:23.887354 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.887318    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hlv8\" (UniqueName: \"kubernetes.io/projected/9ae42a94-d32c-4653-ba9e-6701dc26423f-kube-api-access-4hlv8\") pod \"authorino-557f876f69-nw47h\" (UID: \"9ae42a94-d32c-4653-ba9e-6701dc26423f\") " pod="kuadrant-system/authorino-557f876f69-nw47h"
Sep 10 19:18:23.967504 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.967471    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bpcq2"]
Sep 10 19:18:23.967787 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:18:23.967767    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fjq7n tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-bpcq2" podUID="c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7"
Sep 10 19:18:23.987995 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.987932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hlv8\" (UniqueName: \"kubernetes.io/projected/9ae42a94-d32c-4653-ba9e-6701dc26423f-kube-api-access-4hlv8\") pod \"authorino-557f876f69-nw47h\" (UID: \"9ae42a94-d32c-4653-ba9e-6701dc26423f\") " pod="kuadrant-system/authorino-557f876f69-nw47h"
Sep 10 19:18:23.988114 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.988026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-tls-cert\") pod \"authorino-6d4dfc649f-bpcq2\" (UID: \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\") " pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:23.988114 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:23.988065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjq7n\" (UniqueName: \"kubernetes.io/projected/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-kube-api-access-fjq7n\") pod \"authorino-6d4dfc649f-bpcq2\" (UID: \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\") " pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:24.025678 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.025643    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hlv8\" (UniqueName: \"kubernetes.io/projected/9ae42a94-d32c-4653-ba9e-6701dc26423f-kube-api-access-4hlv8\") pod \"authorino-557f876f69-nw47h\" (UID: \"9ae42a94-d32c-4653-ba9e-6701dc26423f\") " pod="kuadrant-system/authorino-557f876f69-nw47h"
Sep 10 19:18:24.088905 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.088872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-tls-cert\") pod \"authorino-6d4dfc649f-bpcq2\" (UID: \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\") " pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:24.088905 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.088910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjq7n\" (UniqueName: \"kubernetes.io/projected/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-kube-api-access-fjq7n\") pod \"authorino-6d4dfc649f-bpcq2\" (UID: \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\") " pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:24.091429 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.091410    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-tls-cert\") pod \"authorino-6d4dfc649f-bpcq2\" (UID: \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\") " pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:24.102481 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.102460    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjq7n\" (UniqueName: \"kubernetes.io/projected/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-kube-api-access-fjq7n\") pod \"authorino-6d4dfc649f-bpcq2\" (UID: \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\") " pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:24.209551 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.209519    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:24.209908 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.209520    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nw47h"
Sep 10 19:18:24.217196 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.217178    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:24.219824 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.219807    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nw47h"
Sep 10 19:18:24.290877 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.290843    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hlv8\" (UniqueName: \"kubernetes.io/projected/9ae42a94-d32c-4653-ba9e-6701dc26423f-kube-api-access-4hlv8\") pod \"9ae42a94-d32c-4653-ba9e-6701dc26423f\" (UID: \"9ae42a94-d32c-4653-ba9e-6701dc26423f\") "
Sep 10 19:18:24.290877 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.290881    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-tls-cert\") pod \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\" (UID: \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\") "
Sep 10 19:18:24.291138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.290913    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjq7n\" (UniqueName: \"kubernetes.io/projected/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-kube-api-access-fjq7n\") pod \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\" (UID: \"c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7\") "
Sep 10 19:18:24.293173 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.293129    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae42a94-d32c-4653-ba9e-6701dc26423f-kube-api-access-4hlv8" (OuterVolumeSpecName: "kube-api-access-4hlv8") pod "9ae42a94-d32c-4653-ba9e-6701dc26423f" (UID: "9ae42a94-d32c-4653-ba9e-6701dc26423f"). InnerVolumeSpecName "kube-api-access-4hlv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:18:24.293302 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.293224    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-kube-api-access-fjq7n" (OuterVolumeSpecName: "kube-api-access-fjq7n") pod "c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7" (UID: "c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7"). InnerVolumeSpecName "kube-api-access-fjq7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:18:24.293302 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.293245    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7" (UID: "c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:18:24.392010 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.391969    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hlv8\" (UniqueName: \"kubernetes.io/projected/9ae42a94-d32c-4653-ba9e-6701dc26423f-kube-api-access-4hlv8\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:18:24.392010 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.392004    2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-tls-cert\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:18:24.392010 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:24.392014    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjq7n\" (UniqueName: \"kubernetes.io/projected/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7-kube-api-access-fjq7n\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:18:25.214114 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:25.214032    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-nw47h"
Sep 10 19:18:25.214501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:25.214032    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-bpcq2"
Sep 10 19:18:25.248811 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:25.248775    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-nw47h"]
Sep 10 19:18:25.253163 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:25.253122    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-nw47h"]
Sep 10 19:18:25.275347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:25.275317    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bpcq2"]
Sep 10 19:18:25.278894 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:25.278868    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-bpcq2"]
Sep 10 19:18:26.834006 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:26.833922    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae42a94-d32c-4653-ba9e-6701dc26423f" path="/var/lib/kubelet/pods/9ae42a94-d32c-4653-ba9e-6701dc26423f/volumes"
Sep 10 19:18:26.834371 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:26.834280    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7" path="/var/lib/kubelet/pods/c6b21a6d-e083-4cfe-bafd-2c81e7d5fba7/volumes"
Sep 10 19:18:27.227205 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:27.227107    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-g2hn2"]
Sep 10 19:18:27.227367 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:27.227311    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-g2hn2" podUID="0183005d-b99d-49c4-8b35-fd8e2040c817" containerName="authorino" containerID="cri-o://5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6" gracePeriod=30
Sep 10 19:18:27.519412 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:27.519385    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-g2hn2"
Sep 10 19:18:27.619540 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:27.619501    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvc6n\" (UniqueName: \"kubernetes.io/projected/0183005d-b99d-49c4-8b35-fd8e2040c817-kube-api-access-jvc6n\") pod \"0183005d-b99d-49c4-8b35-fd8e2040c817\" (UID: \"0183005d-b99d-49c4-8b35-fd8e2040c817\") "
Sep 10 19:18:27.621789 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:27.621751    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0183005d-b99d-49c4-8b35-fd8e2040c817-kube-api-access-jvc6n" (OuterVolumeSpecName: "kube-api-access-jvc6n") pod "0183005d-b99d-49c4-8b35-fd8e2040c817" (UID: "0183005d-b99d-49c4-8b35-fd8e2040c817"). InnerVolumeSpecName "kube-api-access-jvc6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:18:27.720659 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:27.720623    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvc6n\" (UniqueName: \"kubernetes.io/projected/0183005d-b99d-49c4-8b35-fd8e2040c817-kube-api-access-jvc6n\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:18:28.228965 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.228934    2573 generic.go:356] "Generic (PLEG): container finished" podID="0183005d-b99d-49c4-8b35-fd8e2040c817" containerID="5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6" exitCode=0
Sep 10 19:18:28.229390 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.228982    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-g2hn2"
Sep 10 19:18:28.229390 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.229016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-g2hn2" event={"ID":"0183005d-b99d-49c4-8b35-fd8e2040c817","Type":"ContainerDied","Data":"5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6"}
Sep 10 19:18:28.229390 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.229057    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-g2hn2" event={"ID":"0183005d-b99d-49c4-8b35-fd8e2040c817","Type":"ContainerDied","Data":"3ce0eb2a907309905286f6fa8814e79feda51fa33568d3a039bf6d762acd792f"}
Sep 10 19:18:28.229390 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.229075    2573 scope.go:117] "RemoveContainer" containerID="5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6"
Sep 10 19:18:28.238051 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.238037    2573 scope.go:117] "RemoveContainer" containerID="5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6"
Sep 10 19:18:28.238385 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:18:28.238364    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6\": container with ID starting with 5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6 not found: ID does not exist" containerID="5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6"
Sep 10 19:18:28.238450 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.238393    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6"} err="failed to get container status \"5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6\": rpc error: code = NotFound desc = could not find container \"5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6\": container with ID starting with 5006e587c14fc0078e1e843ae9789faaff8f7f11e061eb36daa7190e2ae24bf6 not found: ID does not exist"
Sep 10 19:18:28.250492 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.250468    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-g2hn2"]
Sep 10 19:18:28.258183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.256998    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-g2hn2"]
Sep 10 19:18:28.833478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:18:28.833443    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0183005d-b99d-49c4-8b35-fd8e2040c817" path="/var/lib/kubelet/pods/0183005d-b99d-49c4-8b35-fd8e2040c817/volumes"
Sep 10 19:19:24.425096 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.425061    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"]
Sep 10 19:19:24.425696 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.425679    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0183005d-b99d-49c4-8b35-fd8e2040c817" containerName="authorino"
Sep 10 19:19:24.425749 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.425700    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0183005d-b99d-49c4-8b35-fd8e2040c817" containerName="authorino"
Sep 10 19:19:24.425812 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.425800    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0183005d-b99d-49c4-8b35-fd8e2040c817" containerName="authorino"
Sep 10 19:19:24.429003 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.428985    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.431524 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.431506    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 19:19:24.432746 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.432714    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xvnhg\""
Sep 10 19:19:24.432839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.432799    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 19:19:24.433033 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.433017    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 19:19:24.440834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.440810    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"]
Sep 10 19:19:24.527503 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.527471    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93b1456-57eb-48fc-b436-db4656447a30-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.527503 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.527503    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.527736 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.527539    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.527736 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.527626    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.527736 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.527648    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8p67\" (UniqueName: \"kubernetes.io/projected/e93b1456-57eb-48fc-b436-db4656447a30-kube-api-access-l8p67\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.527736 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.527684    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.628663 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.628628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93b1456-57eb-48fc-b436-db4656447a30-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.628663 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.628665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.628896 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.628688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.628896 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.628759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.628896 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.628788    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8p67\" (UniqueName: \"kubernetes.io/projected/e93b1456-57eb-48fc-b436-db4656447a30-kube-api-access-l8p67\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.628896 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.628838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.629087 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.629051    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.629143 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.629097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.629297 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.629269    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.631102 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.631076    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e93b1456-57eb-48fc-b436-db4656447a30-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.631349 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.631333    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93b1456-57eb-48fc-b436-db4656447a30-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.636966 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.636948    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8p67\" (UniqueName: \"kubernetes.io/projected/e93b1456-57eb-48fc-b436-db4656447a30-kube-api-access-l8p67\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj\" (UID: \"e93b1456-57eb-48fc-b436-db4656447a30\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:24.740068 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:24.739977    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:25.011590 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:25.011564    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"]
Sep 10 19:19:25.013971 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:19:25.013946    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93b1456_57eb_48fc_b436_db4656447a30.slice/crio-5600fc708de6285689c3f6d81a5327d75fc70f05ae389f9b777f5199812a7d43 WatchSource:0}: Error finding container 5600fc708de6285689c3f6d81a5327d75fc70f05ae389f9b777f5199812a7d43: Status 404 returned error can't find the container with id 5600fc708de6285689c3f6d81a5327d75fc70f05ae389f9b777f5199812a7d43
Sep 10 19:19:25.505769 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:25.505728    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj" event={"ID":"e93b1456-57eb-48fc-b436-db4656447a30","Type":"ContainerStarted","Data":"5600fc708de6285689c3f6d81a5327d75fc70f05ae389f9b777f5199812a7d43"}
Sep 10 19:19:26.512414 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:26.512327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj" event={"ID":"e93b1456-57eb-48fc-b436-db4656447a30","Type":"ContainerStarted","Data":"5689f337b58c11b6ebcdc540b0f5e89a569f90d49ae1ed5cc525eab5b32ff0d1"}
Sep 10 19:19:26.530049 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:26.529993    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj" podStartSLOduration=1.394913867 podStartE2EDuration="2.529977017s" totalImagesPullingTime="1.13506315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:19:26.529387131 +0000 UTC m=+502.189184326" watchObservedRunningTime="2026-09-10 19:19:26.529977017 +0000 UTC m=+502.189774210"
Sep 10 19:19:34.740197 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:34.740133    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:34.740808 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:34.740238    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:34.752885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:34.752859    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:35.567360 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.567328    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj"
Sep 10 19:19:35.802896 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.802866    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"]
Sep 10 19:19:35.806539 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.806522    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.809275 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.809252    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 19:19:35.818130 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.818077    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"]
Sep 10 19:19:35.836801 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.836764    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zqcz\" (UniqueName: \"kubernetes.io/projected/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-kube-api-access-4zqcz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.836988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.836824    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.836988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.836850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.836988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.836903    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.836988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.836945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.836988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.836973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.937945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.937906    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zqcz\" (UniqueName: \"kubernetes.io/projected/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-kube-api-access-4zqcz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.938138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.938002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.938138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.938039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.938138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.938075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.938138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.938124    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.938384 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.938190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.938596 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.938564    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.938723 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.938607    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.938723 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.938623    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.940531 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.940510    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.940772 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.940754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:35.945777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:35.945757    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zqcz\" (UniqueName: \"kubernetes.io/projected/f8c2ebf5-befd-44f2-bac6-e1b9cbddec69-kube-api-access-4zqcz\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dwfp7\" (UID: \"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:36.118115 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:36.118018    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:36.390385 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:36.390304    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"]
Sep 10 19:19:36.394457 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:19:36.394417    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c2ebf5_befd_44f2_bac6_e1b9cbddec69.slice/crio-ee7626fb3aa1da7b773a1f86f7d560ab77eca4d46a1f8dffe14aeddab7be0696 WatchSource:0}: Error finding container ee7626fb3aa1da7b773a1f86f7d560ab77eca4d46a1f8dffe14aeddab7be0696: Status 404 returned error can't find the container with id ee7626fb3aa1da7b773a1f86f7d560ab77eca4d46a1f8dffe14aeddab7be0696
Sep 10 19:19:36.562022 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:36.561985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7" event={"ID":"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69","Type":"ContainerStarted","Data":"ee7626fb3aa1da7b773a1f86f7d560ab77eca4d46a1f8dffe14aeddab7be0696"}
Sep 10 19:19:37.568752 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:37.568715    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7" event={"ID":"f8c2ebf5-befd-44f2-bac6-e1b9cbddec69","Type":"ContainerStarted","Data":"ea098549771fa151e36e3ca7726a2666c612f8219d45032fd2a7b7f46d2d83ae"}
Sep 10 19:19:37.589082 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:37.589026    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7" podStartSLOduration=2.39816231 podStartE2EDuration="2.589009498s" totalImagesPullingTime="190.847188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:19:37.587305611 +0000 UTC m=+513.247102808" watchObservedRunningTime="2026-09-10 19:19:37.589009498 +0000 UTC m=+513.248806690"
Sep 10 19:19:46.118800 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:46.118759    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:46.119272 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:46.118822    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:46.131056 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:46.131032    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:19:46.625995 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:19:46.625966    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dwfp7"
Sep 10 19:21:40.622254 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.622220    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cd585ddf7-k7xjg"]
Sep 10 19:21:40.626028 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.626009    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd585ddf7-k7xjg"
Sep 10 19:21:40.629537 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.629514    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 19:21:40.629654 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.629553    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-77n47\""
Sep 10 19:21:40.634040 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.633616    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cd585ddf7-k7xjg"]
Sep 10 19:21:40.704121 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.704080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/511a22b8-cc23-453d-8b2a-6837afdf8fec-tls-cert\") pod \"authorino-cd585ddf7-k7xjg\" (UID: \"511a22b8-cc23-453d-8b2a-6837afdf8fec\") " pod="kuadrant-system/authorino-cd585ddf7-k7xjg"
Sep 10 19:21:40.704321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.704130    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mzs\" (UniqueName: \"kubernetes.io/projected/511a22b8-cc23-453d-8b2a-6837afdf8fec-kube-api-access-c4mzs\") pod \"authorino-cd585ddf7-k7xjg\" (UID: \"511a22b8-cc23-453d-8b2a-6837afdf8fec\") " pod="kuadrant-system/authorino-cd585ddf7-k7xjg"
Sep 10 19:21:40.805525 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.805477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/511a22b8-cc23-453d-8b2a-6837afdf8fec-tls-cert\") pod \"authorino-cd585ddf7-k7xjg\" (UID: \"511a22b8-cc23-453d-8b2a-6837afdf8fec\") " pod="kuadrant-system/authorino-cd585ddf7-k7xjg"
Sep 10 19:21:40.805525 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.805533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mzs\" (UniqueName: \"kubernetes.io/projected/511a22b8-cc23-453d-8b2a-6837afdf8fec-kube-api-access-c4mzs\") pod \"authorino-cd585ddf7-k7xjg\" (UID: \"511a22b8-cc23-453d-8b2a-6837afdf8fec\") " pod="kuadrant-system/authorino-cd585ddf7-k7xjg"
Sep 10 19:21:40.808204 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.808177    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/511a22b8-cc23-453d-8b2a-6837afdf8fec-tls-cert\") pod \"authorino-cd585ddf7-k7xjg\" (UID: \"511a22b8-cc23-453d-8b2a-6837afdf8fec\") " pod="kuadrant-system/authorino-cd585ddf7-k7xjg"
Sep 10 19:21:40.814349 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.814324    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mzs\" (UniqueName: \"kubernetes.io/projected/511a22b8-cc23-453d-8b2a-6837afdf8fec-kube-api-access-c4mzs\") pod \"authorino-cd585ddf7-k7xjg\" (UID: \"511a22b8-cc23-453d-8b2a-6837afdf8fec\") " pod="kuadrant-system/authorino-cd585ddf7-k7xjg"
Sep 10 19:21:40.936638 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:40.936555    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd585ddf7-k7xjg"
Sep 10 19:21:41.211889 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:41.211864    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cd585ddf7-k7xjg"]
Sep 10 19:21:41.213727 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:21:41.213696    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod511a22b8_cc23_453d_8b2a_6837afdf8fec.slice/crio-cfbaac1717ef0e92dd7f4bba80c9b898578565fac58a86782448d29c352b91ab WatchSource:0}: Error finding container cfbaac1717ef0e92dd7f4bba80c9b898578565fac58a86782448d29c352b91ab: Status 404 returned error can't find the container with id cfbaac1717ef0e92dd7f4bba80c9b898578565fac58a86782448d29c352b91ab
Sep 10 19:21:42.185597 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:42.185556    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd585ddf7-k7xjg" event={"ID":"511a22b8-cc23-453d-8b2a-6837afdf8fec","Type":"ContainerStarted","Data":"b89ecdf66fe3f5b89e68e764bac2dc512773bedd5db442f0c327b2bdc90a2029"}
Sep 10 19:21:42.186003 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:42.185613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd585ddf7-k7xjg" event={"ID":"511a22b8-cc23-453d-8b2a-6837afdf8fec","Type":"ContainerStarted","Data":"cfbaac1717ef0e92dd7f4bba80c9b898578565fac58a86782448d29c352b91ab"}
Sep 10 19:21:42.202543 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:21:42.202480    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-cd585ddf7-k7xjg" podStartSLOduration=1.314980014 podStartE2EDuration="2.202459783s" totalImagesPullingTime="887.479769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:42.202008396 +0000 UTC m=+637.861805590" watchObservedRunningTime="2026-09-10 19:21:42.202459783 +0000 UTC m=+637.862256978"
Sep 10 19:22:01.819350 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:01.819316    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7537-d69f7e"]
Sep 10 19:22:01.822803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:01.822787    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-d69f7e"
Sep 10 19:22:01.826283 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:01.826260    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sbhkd\""
Sep 10 19:22:01.828277 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:01.828255    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-d69f7e"]
Sep 10 19:22:02.005379 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:02.005342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7g7b\" (UniqueName: \"kubernetes.io/projected/d4cebf0f-8bd5-4276-a5d3-178adcc34f39-kube-api-access-b7g7b\") pod \"test-curl-7537-d69f7e\" (UID: \"d4cebf0f-8bd5-4276-a5d3-178adcc34f39\") " pod="opendatahub/test-curl-7537-d69f7e"
Sep 10 19:22:02.106950 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:02.106865    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7g7b\" (UniqueName: \"kubernetes.io/projected/d4cebf0f-8bd5-4276-a5d3-178adcc34f39-kube-api-access-b7g7b\") pod \"test-curl-7537-d69f7e\" (UID: \"d4cebf0f-8bd5-4276-a5d3-178adcc34f39\") " pod="opendatahub/test-curl-7537-d69f7e"
Sep 10 19:22:02.117487 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:02.117459    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7g7b\" (UniqueName: \"kubernetes.io/projected/d4cebf0f-8bd5-4276-a5d3-178adcc34f39-kube-api-access-b7g7b\") pod \"test-curl-7537-d69f7e\" (UID: \"d4cebf0f-8bd5-4276-a5d3-178adcc34f39\") " pod="opendatahub/test-curl-7537-d69f7e"
Sep 10 19:22:02.133117 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:02.133074    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-d69f7e"
Sep 10 19:22:02.405372 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:02.405347    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-d69f7e"]
Sep 10 19:22:02.406999 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:22:02.406975    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4cebf0f_8bd5_4276_a5d3_178adcc34f39.slice/crio-c8354f21ea2d91266faad38841f5afa9e2c3c7a482c5927a4d259b254d6ffafd WatchSource:0}: Error finding container c8354f21ea2d91266faad38841f5afa9e2c3c7a482c5927a4d259b254d6ffafd: Status 404 returned error can't find the container with id c8354f21ea2d91266faad38841f5afa9e2c3c7a482c5927a4d259b254d6ffafd
Sep 10 19:22:03.291999 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:03.291963    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-d69f7e" event={"ID":"d4cebf0f-8bd5-4276-a5d3-178adcc34f39","Type":"ContainerStarted","Data":"c8354f21ea2d91266faad38841f5afa9e2c3c7a482c5927a4d259b254d6ffafd"}
Sep 10 19:22:05.303663 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:05.303625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-d69f7e" event={"ID":"d4cebf0f-8bd5-4276-a5d3-178adcc34f39","Type":"ContainerStarted","Data":"8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee"}
Sep 10 19:22:05.321872 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:05.321814    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7537-d69f7e" podStartSLOduration=2.139405882 podStartE2EDuration="4.321794436s" totalImagesPullingTime="2.182388554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:05.319638536 +0000 UTC m=+660.979435741" watchObservedRunningTime="2026-09-10 19:22:05.321794436 +0000 UTC m=+660.981591632"
Sep 10 19:22:05.925605 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:05.925563    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7537-d69f7e"]
Sep 10 19:22:05.930880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:05.930854    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7537-d69f7e"]
Sep 10 19:22:06.051523 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.051494    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7537-316b1e"]
Sep 10 19:22:06.051912 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.051897    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4cebf0f-8bd5-4276-a5d3-178adcc34f39" containerName="test-curl-7537-d69f7e"
Sep 10 19:22:06.051964 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.051913    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4cebf0f-8bd5-4276-a5d3-178adcc34f39" containerName="test-curl-7537-d69f7e"
Sep 10 19:22:06.052027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.051990    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4cebf0f-8bd5-4276-a5d3-178adcc34f39" containerName="test-curl-7537-d69f7e"
Sep 10 19:22:06.055069 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.055054    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-316b1e"
Sep 10 19:22:06.062204 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.062161    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-316b1e"]
Sep 10 19:22:06.248173 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.248114    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k79vv\" (UniqueName: \"kubernetes.io/projected/a7e3098e-4186-4157-a41a-a507c053e6cd-kube-api-access-k79vv\") pod \"test-curl-7537-316b1e\" (UID: \"a7e3098e-4186-4157-a41a-a507c053e6cd\") " pod="opendatahub/test-curl-7537-316b1e"
Sep 10 19:22:06.308199 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:06.308096    2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7537-d69f7e" container="test-curl-7537-d69f7e"
Sep 10 19:22:06.349717 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.349680    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k79vv\" (UniqueName: \"kubernetes.io/projected/a7e3098e-4186-4157-a41a-a507c053e6cd-kube-api-access-k79vv\") pod \"test-curl-7537-316b1e\" (UID: \"a7e3098e-4186-4157-a41a-a507c053e6cd\") " pod="opendatahub/test-curl-7537-316b1e"
Sep 10 19:22:06.363897 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.363872    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k79vv\" (UniqueName: \"kubernetes.io/projected/a7e3098e-4186-4157-a41a-a507c053e6cd-kube-api-access-k79vv\") pod \"test-curl-7537-316b1e\" (UID: \"a7e3098e-4186-4157-a41a-a507c053e6cd\") " pod="opendatahub/test-curl-7537-316b1e"
Sep 10 19:22:06.364741 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.364719    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-316b1e"
Sep 10 19:22:06.635774 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:06.635750    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7537-316b1e"]
Sep 10 19:22:06.637690 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:22:06.637664    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7e3098e_4186_4157_a41a_a507c053e6cd.slice/crio-8f8f3b04f3cbe1312b8939c5e479afe68bfbe60c776ad0851cac28c54104beb3 WatchSource:0}: Error finding container 8f8f3b04f3cbe1312b8939c5e479afe68bfbe60c776ad0851cac28c54104beb3: Status 404 returned error can't find the container with id 8f8f3b04f3cbe1312b8939c5e479afe68bfbe60c776ad0851cac28c54104beb3
Sep 10 19:22:07.314754 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:07.314721    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-316b1e" event={"ID":"a7e3098e-4186-4157-a41a-a507c053e6cd","Type":"ContainerStarted","Data":"6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506"}
Sep 10 19:22:07.314754 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:07.314748    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7537-d69f7e" podUID="d4cebf0f-8bd5-4276-a5d3-178adcc34f39" containerName="test-curl-7537-d69f7e" containerID="cri-o://8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee" gracePeriod=2
Sep 10 19:22:07.315270 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:07.314763    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7537-316b1e" event={"ID":"a7e3098e-4186-4157-a41a-a507c053e6cd","Type":"ContainerStarted","Data":"8f8f3b04f3cbe1312b8939c5e479afe68bfbe60c776ad0851cac28c54104beb3"}
Sep 10 19:22:07.329483 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:07.329429    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7537-316b1e" podStartSLOduration=1.329409797 podStartE2EDuration="1.329409797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:07.328578271 +0000 UTC m=+662.988375463" watchObservedRunningTime="2026-09-10 19:22:07.329409797 +0000 UTC m=+662.989206991"
Sep 10 19:22:07.926454 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:07.926417    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7537-316b1e"]
Sep 10 19:22:07.946857 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:07.946826    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7537-316b1e"]
Sep 10 19:22:08.319851 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:08.319814    2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7537-316b1e" container="test-curl-7537-316b1e"
Sep 10 19:22:09.325003 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:09.324966    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7537-316b1e" podUID="a7e3098e-4186-4157-a41a-a507c053e6cd" containerName="test-curl-7537-316b1e" containerID="cri-o://6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506" gracePeriod=2
Sep 10 19:22:09.472742 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:09.472712    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-d69f7e"
Sep 10 19:22:09.481594 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:09.481575    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7g7b\" (UniqueName: \"kubernetes.io/projected/d4cebf0f-8bd5-4276-a5d3-178adcc34f39-kube-api-access-b7g7b\") pod \"d4cebf0f-8bd5-4276-a5d3-178adcc34f39\" (UID: \"d4cebf0f-8bd5-4276-a5d3-178adcc34f39\") "
Sep 10 19:22:09.483750 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:09.483712    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cebf0f-8bd5-4276-a5d3-178adcc34f39-kube-api-access-b7g7b" (OuterVolumeSpecName: "kube-api-access-b7g7b") pod "d4cebf0f-8bd5-4276-a5d3-178adcc34f39" (UID: "d4cebf0f-8bd5-4276-a5d3-178adcc34f39"). InnerVolumeSpecName "kube-api-access-b7g7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:22:09.583432 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:09.583356    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7g7b\" (UniqueName: \"kubernetes.io/projected/d4cebf0f-8bd5-4276-a5d3-178adcc34f39-kube-api-access-b7g7b\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:22:10.331463 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:10.331430    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-d69f7e"
Sep 10 19:22:10.331945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:10.331387    2573 generic.go:356] "Generic (PLEG): container finished" podID="d4cebf0f-8bd5-4276-a5d3-178adcc34f39" containerID="8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee" exitCode=137
Sep 10 19:22:10.331945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:10.331501    2573 scope.go:117] "RemoveContainer" containerID="8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee"
Sep 10 19:22:10.341902 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:10.341866    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7537-d69f7e\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="d4cebf0f-8bd5-4276-a5d3-178adcc34f39" pod="opendatahub/test-curl-7537-d69f7e"
Sep 10 19:22:10.342141 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:10.342124    2573 scope.go:117] "RemoveContainer" containerID="8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee"
Sep 10 19:22:10.342484 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:10.342460    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee\": container with ID starting with 8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee not found: ID does not exist" containerID="8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee"
Sep 10 19:22:10.342562 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:10.342495    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee"} err="failed to get container status \"8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee\": rpc error: code = NotFound desc = could not find container \"8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee\": container with ID starting with 8a2f55e7a170c48792ab9adfbd85b310289f80b27b22fa52f9f23e31d1c9eeee not found: ID does not exist"
Sep 10 19:22:10.833569 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:10.833536    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4cebf0f-8bd5-4276-a5d3-178adcc34f39" path="/var/lib/kubelet/pods/d4cebf0f-8bd5-4276-a5d3-178adcc34f39/volumes"
Sep 10 19:22:11.532083 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.532054    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-316b1e"
Sep 10 19:22:11.601937 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.601820    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k79vv\" (UniqueName: \"kubernetes.io/projected/a7e3098e-4186-4157-a41a-a507c053e6cd-kube-api-access-k79vv\") pod \"a7e3098e-4186-4157-a41a-a507c053e6cd\" (UID: \"a7e3098e-4186-4157-a41a-a507c053e6cd\") "
Sep 10 19:22:11.604126 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.604087    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e3098e-4186-4157-a41a-a507c053e6cd-kube-api-access-k79vv" (OuterVolumeSpecName: "kube-api-access-k79vv") pod "a7e3098e-4186-4157-a41a-a507c053e6cd" (UID: "a7e3098e-4186-4157-a41a-a507c053e6cd"). InnerVolumeSpecName "kube-api-access-k79vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:22:11.703290 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.703249    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k79vv\" (UniqueName: \"kubernetes.io/projected/a7e3098e-4186-4157-a41a-a507c053e6cd-kube-api-access-k79vv\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:22:11.795897 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.795862    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"]
Sep 10 19:22:11.796317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.796299    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e3098e-4186-4157-a41a-a507c053e6cd" containerName="test-curl-7537-316b1e"
Sep 10 19:22:11.796317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.796314    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e3098e-4186-4157-a41a-a507c053e6cd" containerName="test-curl-7537-316b1e"
Sep 10 19:22:11.796488 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.796418    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e3098e-4186-4157-a41a-a507c053e6cd" containerName="test-curl-7537-316b1e"
Sep 10 19:22:11.799617 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.799597    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:11.802255 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.802229    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8db50f5a-serving-cert\""
Sep 10 19:22:11.802361 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.802334    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 19:22:11.803432 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.803411    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lw7bv\""
Sep 10 19:22:11.813792 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.813763    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"]
Sep 10 19:22:11.904936 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.904825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rln4\" (UniqueName: \"kubernetes.io/projected/1f8cb72e-037e-42de-a96e-1e90943e9479-kube-api-access-9rln4\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:11.904936 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.904880    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-metrics-certs\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:11.905135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.905009    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-maas-api-tls\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:11.905135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:11.905040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cb72e-037e-42de-a96e-1e90943e9479-openshift-service-ca\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.006477 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.006439    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-maas-api-tls\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.006477 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.006477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cb72e-037e-42de-a96e-1e90943e9479-openshift-service-ca\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.006698 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.006522    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rln4\" (UniqueName: \"kubernetes.io/projected/1f8cb72e-037e-42de-a96e-1e90943e9479-kube-api-access-9rln4\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.006698 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.006538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-metrics-certs\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.007330 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.007302    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cb72e-037e-42de-a96e-1e90943e9479-openshift-service-ca\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.009131 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.009099    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-maas-api-tls\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.009131 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.009103    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-metrics-certs\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.016701 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.016681    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rln4\" (UniqueName: \"kubernetes.io/projected/1f8cb72e-037e-42de-a96e-1e90943e9479-kube-api-access-9rln4\") pod \"maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.110829 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.110797    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:12.349969 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.349934    2573 generic.go:356] "Generic (PLEG): container finished" podID="a7e3098e-4186-4157-a41a-a507c053e6cd" containerID="6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506" exitCode=137
Sep 10 19:22:12.350109 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.349983    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7537-316b1e"
Sep 10 19:22:12.350109 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.350040    2573 scope.go:117] "RemoveContainer" containerID="6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506"
Sep 10 19:22:12.358799 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.358780    2573 scope.go:117] "RemoveContainer" containerID="6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506"
Sep 10 19:22:12.359095 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:12.359078    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506\": container with ID starting with 6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506 not found: ID does not exist" containerID="6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506"
Sep 10 19:22:12.359193 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.359102    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506"} err="failed to get container status \"6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506\": rpc error: code = NotFound desc = could not find container \"6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506\": container with ID starting with 6f6daeb1754e09b0b4ff81a024c3234c8eeb75d50713be33b6feec8fd941c506 not found: ID does not exist"
Sep 10 19:22:12.361351 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:12.361328    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7537-316b1e\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="a7e3098e-4186-4157-a41a-a507c053e6cd" pod="opendatahub/test-curl-7537-316b1e"
Sep 10 19:22:12.410622 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.410598    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"]
Sep 10 19:22:12.412988 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:22:12.412924    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8cb72e_037e_42de_a96e_1e90943e9479.slice/crio-88fa985b7f135cd4d01b939641989596adf476b8375bb8b94420333f965f1a30 WatchSource:0}: Error finding container 88fa985b7f135cd4d01b939641989596adf476b8375bb8b94420333f965f1a30: Status 404 returned error can't find the container with id 88fa985b7f135cd4d01b939641989596adf476b8375bb8b94420333f965f1a30
Sep 10 19:22:12.647936 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.647898    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"]
Sep 10 19:22:12.652806 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.652786    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:12.655487 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:12.655463    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7537-316b1e\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="a7e3098e-4186-4157-a41a-a507c053e6cd" pod="opendatahub/test-curl-7537-316b1e"
Sep 10 19:22:12.655984 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.655965    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8db50f5a-dockercfg-bmbzm\""
Sep 10 19:22:12.656585 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.656571    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8db50f5a\""
Sep 10 19:22:12.666934 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.666871    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"]
Sep 10 19:22:12.714001 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.713971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrxl\" (UniqueName: \"kubernetes.io/projected/472d7cd8-99a1-4727-abdf-830a20250122-kube-api-access-wtrxl\") pod \"payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7\" (UID: \"472d7cd8-99a1-4727-abdf-830a20250122\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:12.714209 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.714015    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472d7cd8-99a1-4727-abdf-830a20250122-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7\" (UID: \"472d7cd8-99a1-4727-abdf-830a20250122\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:12.814858 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.814816    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrxl\" (UniqueName: \"kubernetes.io/projected/472d7cd8-99a1-4727-abdf-830a20250122-kube-api-access-wtrxl\") pod \"payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7\" (UID: \"472d7cd8-99a1-4727-abdf-830a20250122\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:12.814858 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.814863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472d7cd8-99a1-4727-abdf-830a20250122-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7\" (UID: \"472d7cd8-99a1-4727-abdf-830a20250122\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:12.815545 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.815525    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472d7cd8-99a1-4727-abdf-830a20250122-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7\" (UID: \"472d7cd8-99a1-4727-abdf-830a20250122\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:12.829678 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.829651    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrxl\" (UniqueName: \"kubernetes.io/projected/472d7cd8-99a1-4727-abdf-830a20250122-kube-api-access-wtrxl\") pod \"payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7\" (UID: \"472d7cd8-99a1-4727-abdf-830a20250122\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:12.836138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.836102    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e3098e-4186-4157-a41a-a507c053e6cd" path="/var/lib/kubelet/pods/a7e3098e-4186-4157-a41a-a507c053e6cd/volumes"
Sep 10 19:22:12.963607 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:12.963514    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:13.258522 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:13.258491    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"]
Sep 10 19:22:13.259886 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:22:13.259858    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472d7cd8_99a1_4727_abdf_830a20250122.slice/crio-1a42a4d2cc5db2e99d36290c781ff97912911c9f22ec039cbd4c762b1a5a6cf6 WatchSource:0}: Error finding container 1a42a4d2cc5db2e99d36290c781ff97912911c9f22ec039cbd4c762b1a5a6cf6: Status 404 returned error can't find the container with id 1a42a4d2cc5db2e99d36290c781ff97912911c9f22ec039cbd4c762b1a5a6cf6
Sep 10 19:22:13.361825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:13.361759    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w" event={"ID":"1f8cb72e-037e-42de-a96e-1e90943e9479","Type":"ContainerStarted","Data":"88fa985b7f135cd4d01b939641989596adf476b8375bb8b94420333f965f1a30"}
Sep 10 19:22:13.364891 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:13.364846    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7" event={"ID":"472d7cd8-99a1-4727-abdf-830a20250122","Type":"ContainerStarted","Data":"1a42a4d2cc5db2e99d36290c781ff97912911c9f22ec039cbd4c762b1a5a6cf6"}
Sep 10 19:22:14.378863 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.378825    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"]
Sep 10 19:22:14.383344 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.383320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:14.392711 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.392277    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-3c7a5e-dockercfg-nnphb\""
Sep 10 19:22:14.393020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.393000    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3c7a5e\""
Sep 10 19:22:14.399698 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.399673    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"]
Sep 10 19:22:14.435624 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.435583    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278c2e2-7b45-41aa-bd26-941c69ff893b-plugins-config-volume\") pod \"payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j\" (UID: \"3278c2e2-7b45-41aa-bd26-941c69ff893b\") " pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:14.435821 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.435637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7hx\" (UniqueName: \"kubernetes.io/projected/3278c2e2-7b45-41aa-bd26-941c69ff893b-kube-api-access-nw7hx\") pod \"payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j\" (UID: \"3278c2e2-7b45-41aa-bd26-941c69ff893b\") " pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:14.536747 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.536712    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278c2e2-7b45-41aa-bd26-941c69ff893b-plugins-config-volume\") pod \"payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j\" (UID: \"3278c2e2-7b45-41aa-bd26-941c69ff893b\") " pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:14.536922 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.536765    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7hx\" (UniqueName: \"kubernetes.io/projected/3278c2e2-7b45-41aa-bd26-941c69ff893b-kube-api-access-nw7hx\") pod \"payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j\" (UID: \"3278c2e2-7b45-41aa-bd26-941c69ff893b\") " pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:14.537629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.537604    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278c2e2-7b45-41aa-bd26-941c69ff893b-plugins-config-volume\") pod \"payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j\" (UID: \"3278c2e2-7b45-41aa-bd26-941c69ff893b\") " pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:14.545937 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.545912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7hx\" (UniqueName: \"kubernetes.io/projected/3278c2e2-7b45-41aa-bd26-941c69ff893b-kube-api-access-nw7hx\") pod \"payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j\" (UID: \"3278c2e2-7b45-41aa-bd26-941c69ff893b\") " pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:14.698916 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:14.698840    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:15.006726 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:15.006705    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"]
Sep 10 19:22:15.008711 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:22:15.008687    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3278c2e2_7b45_41aa_bd26_941c69ff893b.slice/crio-ba65b6f92b4862f9d2d4d09de43ed3bb62c6de880cb5ea26b7e52264386dfa55 WatchSource:0}: Error finding container ba65b6f92b4862f9d2d4d09de43ed3bb62c6de880cb5ea26b7e52264386dfa55: Status 404 returned error can't find the container with id ba65b6f92b4862f9d2d4d09de43ed3bb62c6de880cb5ea26b7e52264386dfa55
Sep 10 19:22:15.378316 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:15.378282    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j" event={"ID":"3278c2e2-7b45-41aa-bd26-941c69ff893b","Type":"ContainerStarted","Data":"ba65b6f92b4862f9d2d4d09de43ed3bb62c6de880cb5ea26b7e52264386dfa55"}
Sep 10 19:22:15.380136 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:15.380100    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w" event={"ID":"1f8cb72e-037e-42de-a96e-1e90943e9479","Type":"ContainerStarted","Data":"a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2"}
Sep 10 19:22:15.380547 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:15.380194    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:15.401068 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:15.401010    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w" podStartSLOduration=2.069941117 podStartE2EDuration="4.400990085s" totalImagesPullingTime="2.331048968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:15.399726155 +0000 UTC m=+671.059523359" watchObservedRunningTime="2026-09-10 19:22:15.400990085 +0000 UTC m=+671.060787300"
Sep 10 19:22:16.388832 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:16.388798    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j" event={"ID":"3278c2e2-7b45-41aa-bd26-941c69ff893b","Type":"ContainerStarted","Data":"63d8ae8bf9e0f663b02b60d238d71152123d015f493ffd598e166db16fbb0c6a"}
Sep 10 19:22:16.389318 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:16.388936    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:16.390697 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:16.390668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7" event={"ID":"472d7cd8-99a1-4727-abdf-830a20250122","Type":"ContainerStarted","Data":"ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752"}
Sep 10 19:22:16.390837 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:16.390811    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:16.409867 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:16.409818    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j" podStartSLOduration=1.199396423 podStartE2EDuration="2.409803977s" totalImagesPullingTime="1.210407554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:16.408538211 +0000 UTC m=+672.068335404" watchObservedRunningTime="2026-09-10 19:22:16.409803977 +0000 UTC m=+672.069601173"
Sep 10 19:22:16.429703 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:16.429652    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7" podStartSLOduration=1.476527808 podStartE2EDuration="4.429632243s" totalImagesPullingTime="2.953104435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:16.427981244 +0000 UTC m=+672.087778439" watchObservedRunningTime="2026-09-10 19:22:16.429632243 +0000 UTC m=+672.089429446"
Sep 10 19:22:21.397246 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:21.397219    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:27.398008 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:27.397971    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:22:27.398472 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:27.398043    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:34.838038 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:34.838006    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"]
Sep 10 19:22:34.844746 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:34.844728    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:34.847327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:34.847302    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b960a2\""
Sep 10 19:22:34.847435 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:34.847359    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b960a2-dockercfg-z45rb\""
Sep 10 19:22:34.852467 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:34.852444    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"]
Sep 10 19:22:34.959011 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:34.958974    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34fbfff-47c6-409d-afe6-8770428d7219-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz\" (UID: \"f34fbfff-47c6-409d-afe6-8770428d7219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:34.959186 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:34.959088    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkd9\" (UniqueName: \"kubernetes.io/projected/f34fbfff-47c6-409d-afe6-8770428d7219-kube-api-access-slkd9\") pod \"payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz\" (UID: \"f34fbfff-47c6-409d-afe6-8770428d7219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:35.060341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:35.060310    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34fbfff-47c6-409d-afe6-8770428d7219-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz\" (UID: \"f34fbfff-47c6-409d-afe6-8770428d7219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:35.060530 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:35.060388    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkd9\" (UniqueName: \"kubernetes.io/projected/f34fbfff-47c6-409d-afe6-8770428d7219-kube-api-access-slkd9\") pod \"payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz\" (UID: \"f34fbfff-47c6-409d-afe6-8770428d7219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:35.061020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:35.060998    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34fbfff-47c6-409d-afe6-8770428d7219-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz\" (UID: \"f34fbfff-47c6-409d-afe6-8770428d7219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:35.070529 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:35.070501    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkd9\" (UniqueName: \"kubernetes.io/projected/f34fbfff-47c6-409d-afe6-8770428d7219-kube-api-access-slkd9\") pod \"payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz\" (UID: \"f34fbfff-47c6-409d-afe6-8770428d7219\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:35.156738 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:35.156658    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:35.434581 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:35.434422    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"]
Sep 10 19:22:35.437478 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:22:35.437447    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34fbfff_47c6_409d_afe6_8770428d7219.slice/crio-e697591971b45a35c411300023a102946aa170ad697844f58372d4c90f01ec4c WatchSource:0}: Error finding container e697591971b45a35c411300023a102946aa170ad697844f58372d4c90f01ec4c: Status 404 returned error can't find the container with id e697591971b45a35c411300023a102946aa170ad697844f58372d4c90f01ec4c
Sep 10 19:22:35.493030 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:35.493003    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz" event={"ID":"f34fbfff-47c6-409d-afe6-8770428d7219","Type":"ContainerStarted","Data":"e697591971b45a35c411300023a102946aa170ad697844f58372d4c90f01ec4c"}
Sep 10 19:22:36.500203 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:36.500136    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz" event={"ID":"f34fbfff-47c6-409d-afe6-8770428d7219","Type":"ContainerStarted","Data":"244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c"}
Sep 10 19:22:36.500622 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:36.500237    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:36.518396 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:36.518335    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz" podStartSLOduration=2.518312922 podStartE2EDuration="2.518312922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:36.517404903 +0000 UTC m=+692.177202098" watchObservedRunningTime="2026-09-10 19:22:36.518312922 +0000 UTC m=+692.178110116"
Sep 10 19:22:47.507410 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:47.507374    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:22:52.798448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:52.798414    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"]
Sep 10 19:22:52.798837 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:52.798683    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w" podUID="1f8cb72e-037e-42de-a96e-1e90943e9479" containerName="maas-api" containerID="cri-o://a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2" gracePeriod=30
Sep 10 19:22:52.936181 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:52.936135    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"]
Sep 10 19:22:52.936405 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:52.936380    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7" podUID="472d7cd8-99a1-4727-abdf-830a20250122" containerName="payload-pre-processing" containerID="cri-o://ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752" gracePeriod=30
Sep 10 19:22:53.062612 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.062586    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:53.133733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.133682    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rln4\" (UniqueName: \"kubernetes.io/projected/1f8cb72e-037e-42de-a96e-1e90943e9479-kube-api-access-9rln4\") pod \"1f8cb72e-037e-42de-a96e-1e90943e9479\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") "
Sep 10 19:22:53.133733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.133728    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-metrics-certs\") pod \"1f8cb72e-037e-42de-a96e-1e90943e9479\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") "
Sep 10 19:22:53.133934 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.133850    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-maas-api-tls\") pod \"1f8cb72e-037e-42de-a96e-1e90943e9479\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") "
Sep 10 19:22:53.133934 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.133884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cb72e-037e-42de-a96e-1e90943e9479-openshift-service-ca\") pod \"1f8cb72e-037e-42de-a96e-1e90943e9479\" (UID: \"1f8cb72e-037e-42de-a96e-1e90943e9479\") "
Sep 10 19:22:53.134638 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.134522    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8cb72e-037e-42de-a96e-1e90943e9479-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f8cb72e-037e-42de-a96e-1e90943e9479" (UID: "1f8cb72e-037e-42de-a96e-1e90943e9479"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:22:53.136795 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.136766    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f8cb72e-037e-42de-a96e-1e90943e9479" (UID: "1f8cb72e-037e-42de-a96e-1e90943e9479"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:22:53.136887 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.136864    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8cb72e-037e-42de-a96e-1e90943e9479-kube-api-access-9rln4" (OuterVolumeSpecName: "kube-api-access-9rln4") pod "1f8cb72e-037e-42de-a96e-1e90943e9479" (UID: "1f8cb72e-037e-42de-a96e-1e90943e9479"). InnerVolumeSpecName "kube-api-access-9rln4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:22:53.137124 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.137101    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f8cb72e-037e-42de-a96e-1e90943e9479" (UID: "1f8cb72e-037e-42de-a96e-1e90943e9479"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:22:53.188209 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.188183    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:53.236184 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.235613    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtrxl\" (UniqueName: \"kubernetes.io/projected/472d7cd8-99a1-4727-abdf-830a20250122-kube-api-access-wtrxl\") pod \"472d7cd8-99a1-4727-abdf-830a20250122\" (UID: \"472d7cd8-99a1-4727-abdf-830a20250122\") "
Sep 10 19:22:53.236184 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.235657    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472d7cd8-99a1-4727-abdf-830a20250122-plugins-config-volume\") pod \"472d7cd8-99a1-4727-abdf-830a20250122\" (UID: \"472d7cd8-99a1-4727-abdf-830a20250122\") "
Sep 10 19:22:53.236184 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.236000    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8cb72e-037e-42de-a96e-1e90943e9479-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:22:53.236184 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.236018    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rln4\" (UniqueName: \"kubernetes.io/projected/1f8cb72e-037e-42de-a96e-1e90943e9479-kube-api-access-9rln4\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:22:53.236184 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.236033    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:22:53.236184 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.236047    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f8cb72e-037e-42de-a96e-1e90943e9479-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:22:53.236949 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.236906    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472d7cd8-99a1-4727-abdf-830a20250122-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "472d7cd8-99a1-4727-abdf-830a20250122" (UID: "472d7cd8-99a1-4727-abdf-830a20250122"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:22:53.239904 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.239865    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472d7cd8-99a1-4727-abdf-830a20250122-kube-api-access-wtrxl" (OuterVolumeSpecName: "kube-api-access-wtrxl") pod "472d7cd8-99a1-4727-abdf-830a20250122" (UID: "472d7cd8-99a1-4727-abdf-830a20250122"). InnerVolumeSpecName "kube-api-access-wtrxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:22:53.337092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.337004    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrxl\" (UniqueName: \"kubernetes.io/projected/472d7cd8-99a1-4727-abdf-830a20250122-kube-api-access-wtrxl\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:22:53.337092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.337032    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/472d7cd8-99a1-4727-abdf-830a20250122-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:22:53.591651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.591557    2573 generic.go:356] "Generic (PLEG): container finished" podID="472d7cd8-99a1-4727-abdf-830a20250122" containerID="ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752" exitCode=0
Sep 10 19:22:53.591651 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.591604    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"
Sep 10 19:22:53.591866 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.591648    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7" event={"ID":"472d7cd8-99a1-4727-abdf-830a20250122","Type":"ContainerDied","Data":"ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752"}
Sep 10 19:22:53.591866 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.591701    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7" event={"ID":"472d7cd8-99a1-4727-abdf-830a20250122","Type":"ContainerDied","Data":"1a42a4d2cc5db2e99d36290c781ff97912911c9f22ec039cbd4c762b1a5a6cf6"}
Sep 10 19:22:53.591866 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.591752    2573 scope.go:117] "RemoveContainer" containerID="ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752"
Sep 10 19:22:53.593174 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.593122    2573 generic.go:356] "Generic (PLEG): container finished" podID="1f8cb72e-037e-42de-a96e-1e90943e9479" containerID="a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2" exitCode=0
Sep 10 19:22:53.593357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.593179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w" event={"ID":"1f8cb72e-037e-42de-a96e-1e90943e9479","Type":"ContainerDied","Data":"a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2"}
Sep 10 19:22:53.593357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.593223    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w" event={"ID":"1f8cb72e-037e-42de-a96e-1e90943e9479","Type":"ContainerDied","Data":"88fa985b7f135cd4d01b939641989596adf476b8375bb8b94420333f965f1a30"}
Sep 10 19:22:53.593357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.593273    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"
Sep 10 19:22:53.600877 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.600859    2573 scope.go:117] "RemoveContainer" containerID="ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752"
Sep 10 19:22:53.601213 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:53.601188    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752\": container with ID starting with ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752 not found: ID does not exist" containerID="ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752"
Sep 10 19:22:53.601303 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.601225    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752"} err="failed to get container status \"ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752\": rpc error: code = NotFound desc = could not find container \"ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752\": container with ID starting with ca7eab06e05aa477c710a84656e91538901403f8fb011ae2ac0b60859f8c8752 not found: ID does not exist"
Sep 10 19:22:53.601303 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.601248    2573 scope.go:117] "RemoveContainer" containerID="a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2"
Sep 10 19:22:53.610026 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.610007    2573 scope.go:117] "RemoveContainer" containerID="a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2"
Sep 10 19:22:53.610316 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:53.610299    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2\": container with ID starting with a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2 not found: ID does not exist" containerID="a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2"
Sep 10 19:22:53.610377 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.610326    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2"} err="failed to get container status \"a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2\": rpc error: code = NotFound desc = could not find container \"a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2\": container with ID starting with a13a0a9581dc3bc63d45a459ac0b0fa509a38de208c92b12e9ae9e19c006e5f2 not found: ID does not exist"
Sep 10 19:22:53.621889 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.621865    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"]
Sep 10 19:22:53.627362 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.627341    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8db50f5a-5585c8d8cd-2g5c7"]
Sep 10 19:22:53.640212 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.640190    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"]
Sep 10 19:22:53.648848 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:53.648819    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8db50f5a-6c76dbc5b9-8l66w"]
Sep 10 19:22:54.834317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:54.834279    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f8cb72e-037e-42de-a96e-1e90943e9479" path="/var/lib/kubelet/pods/1f8cb72e-037e-42de-a96e-1e90943e9479/volumes"
Sep 10 19:22:54.834807 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:54.834792    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="472d7cd8-99a1-4727-abdf-830a20250122" path="/var/lib/kubelet/pods/472d7cd8-99a1-4727-abdf-830a20250122/volumes"
Sep 10 19:22:57.400876 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.400843    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"]
Sep 10 19:22:57.401474 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.401458    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472d7cd8-99a1-4727-abdf-830a20250122" containerName="payload-pre-processing"
Sep 10 19:22:57.401521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.401478    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="472d7cd8-99a1-4727-abdf-830a20250122" containerName="payload-pre-processing"
Sep 10 19:22:57.401521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.401504    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f8cb72e-037e-42de-a96e-1e90943e9479" containerName="maas-api"
Sep 10 19:22:57.401521 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.401513    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f8cb72e-037e-42de-a96e-1e90943e9479" containerName="maas-api"
Sep 10 19:22:57.401632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.401621    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f8cb72e-037e-42de-a96e-1e90943e9479" containerName="maas-api"
Sep 10 19:22:57.401667 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.401636    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="472d7cd8-99a1-4727-abdf-830a20250122" containerName="payload-pre-processing"
Sep 10 19:22:57.406621 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.406602    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.410288 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.410197    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b59ba4a1-serving-cert\""
Sep 10 19:22:57.410288 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.410257    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lw7bv\""
Sep 10 19:22:57.410980 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.410961    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 19:22:57.415608 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.415583    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"]
Sep 10 19:22:57.481415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.481383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-openshift-service-ca\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.481602 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.481531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8rj\" (UniqueName: \"kubernetes.io/projected/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-kube-api-access-qx8rj\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.481602 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.481568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-maas-api-tls\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.481602 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.481601    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-metrics-certs\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.583114 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.583077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8rj\" (UniqueName: \"kubernetes.io/projected/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-kube-api-access-qx8rj\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.583344 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.583121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-maas-api-tls\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.583344 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:57.583259    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-serving-cert: secret "maas-api-e2e-migrate-b59ba4a1-serving-cert" not found
Sep 10 19:22:57.583344 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.583320    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-metrics-certs\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.583344 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:22:57.583328    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-maas-api-tls podName:82ddceb0-46f1-4ec7-bef6-a04726fd69cf nodeName:}" failed. No retries permitted until 2026-09-10 19:22:58.083309345 +0000 UTC m=+713.743106548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-maas-api-tls") pod "maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5" (UID: "82ddceb0-46f1-4ec7-bef6-a04726fd69cf") : secret "maas-api-e2e-migrate-b59ba4a1-serving-cert" not found
Sep 10 19:22:57.583595 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.583404    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-openshift-service-ca\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.584023 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.584002    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-openshift-service-ca\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.586087 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.586066    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-metrics-certs\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:57.598402 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:57.598379    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8rj\" (UniqueName: \"kubernetes.io/projected/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-kube-api-access-qx8rj\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:58.089370 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.089333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-maas-api-tls\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:58.092002 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.091968    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-maas-api-tls\") pod \"maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:58.317972 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.317934    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:58.595357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.595328    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"]
Sep 10 19:22:58.597326 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:22:58.597290    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ddceb0_46f1_4ec7_bef6_a04726fd69cf.slice/crio-4bc1d7adbb7b3cd1b0eac000aa301ba439716db46e53c4da7bdb5246a6112f7c WatchSource:0}: Error finding container 4bc1d7adbb7b3cd1b0eac000aa301ba439716db46e53c4da7bdb5246a6112f7c: Status 404 returned error can't find the container with id 4bc1d7adbb7b3cd1b0eac000aa301ba439716db46e53c4da7bdb5246a6112f7c
Sep 10 19:22:58.598674 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.598656    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:22:58.621470 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.621415    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5" event={"ID":"82ddceb0-46f1-4ec7-bef6-a04726fd69cf","Type":"ContainerStarted","Data":"4bc1d7adbb7b3cd1b0eac000aa301ba439716db46e53c4da7bdb5246a6112f7c"}
Sep 10 19:22:58.642483 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.642454    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"]
Sep 10 19:22:58.647091 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.647075    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:22:58.650246 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.650223    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b59ba4a1-dockercfg-rztm2\""
Sep 10 19:22:58.650629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.650614    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b59ba4a1\""
Sep 10 19:22:58.665911 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.665892    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"]
Sep 10 19:22:58.760818 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.760787    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"]
Sep 10 19:22:58.764318 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.764297    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:22:58.789104 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.783759    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"]
Sep 10 19:22:58.797625 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.797592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa86154-3d3c-43ed-acd0-cde0e365906b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6\" (UID: \"dfa86154-3d3c-43ed-acd0-cde0e365906b\") " pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:22:58.797761 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.797690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-888d5\" (UniqueName: \"kubernetes.io/projected/dfa86154-3d3c-43ed-acd0-cde0e365906b-kube-api-access-888d5\") pod \"payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6\" (UID: \"dfa86154-3d3c-43ed-acd0-cde0e365906b\") " pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:22:58.898442 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.898358    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa86154-3d3c-43ed-acd0-cde0e365906b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6\" (UID: \"dfa86154-3d3c-43ed-acd0-cde0e365906b\") " pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:22:58.898442 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.898403    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsc25\" (UniqueName: \"kubernetes.io/projected/50ad5e20-94c2-4467-a7e2-887364c10636-kube-api-access-zsc25\") pod \"payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c\" (UID: \"50ad5e20-94c2-4467-a7e2-887364c10636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:22:58.898677 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.898599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-888d5\" (UniqueName: \"kubernetes.io/projected/dfa86154-3d3c-43ed-acd0-cde0e365906b-kube-api-access-888d5\") pod \"payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6\" (UID: \"dfa86154-3d3c-43ed-acd0-cde0e365906b\") " pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:22:58.898677 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.898647    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad5e20-94c2-4467-a7e2-887364c10636-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c\" (UID: \"50ad5e20-94c2-4467-a7e2-887364c10636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:22:58.899059 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.899041    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa86154-3d3c-43ed-acd0-cde0e365906b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6\" (UID: \"dfa86154-3d3c-43ed-acd0-cde0e365906b\") " pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:22:58.912035 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.912005    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-888d5\" (UniqueName: \"kubernetes.io/projected/dfa86154-3d3c-43ed-acd0-cde0e365906b-kube-api-access-888d5\") pod \"payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6\" (UID: \"dfa86154-3d3c-43ed-acd0-cde0e365906b\") " pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:22:58.957415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.957390    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:22:58.999305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.999267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsc25\" (UniqueName: \"kubernetes.io/projected/50ad5e20-94c2-4467-a7e2-887364c10636-kube-api-access-zsc25\") pod \"payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c\" (UID: \"50ad5e20-94c2-4467-a7e2-887364c10636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:22:58.999452 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:58.999417    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad5e20-94c2-4467-a7e2-887364c10636-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c\" (UID: \"50ad5e20-94c2-4467-a7e2-887364c10636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:22:59.000082 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.000057    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad5e20-94c2-4467-a7e2-887364c10636-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c\" (UID: \"50ad5e20-94c2-4467-a7e2-887364c10636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:22:59.025561 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.025531    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsc25\" (UniqueName: \"kubernetes.io/projected/50ad5e20-94c2-4467-a7e2-887364c10636-kube-api-access-zsc25\") pod \"payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c\" (UID: \"50ad5e20-94c2-4467-a7e2-887364c10636\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:22:59.074173 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.074121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:22:59.237990 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.237959    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"]
Sep 10 19:22:59.239976 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:22:59.239947    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa86154_3d3c_43ed_acd0_cde0e365906b.slice/crio-eb5b3744abc79d46db485cb2ace4af72947d5b851745abf00075e97782432835 WatchSource:0}: Error finding container eb5b3744abc79d46db485cb2ace4af72947d5b851745abf00075e97782432835: Status 404 returned error can't find the container with id eb5b3744abc79d46db485cb2ace4af72947d5b851745abf00075e97782432835
Sep 10 19:22:59.357187 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.357138    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"]
Sep 10 19:22:59.360495 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:22:59.360462    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ad5e20_94c2_4467_a7e2_887364c10636.slice/crio-ea1c80d2731740788695a542ea9c22be207c9f4a90136870f740c498cf1042a2 WatchSource:0}: Error finding container ea1c80d2731740788695a542ea9c22be207c9f4a90136870f740c498cf1042a2: Status 404 returned error can't find the container with id ea1c80d2731740788695a542ea9c22be207c9f4a90136870f740c498cf1042a2
Sep 10 19:22:59.628663 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.628607    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c" event={"ID":"50ad5e20-94c2-4467-a7e2-887364c10636","Type":"ContainerStarted","Data":"025f6143b0cc6ed17c3fe1e2ddfca67c550f52fbdde0ce41c97f947f42c78225"}
Sep 10 19:22:59.629105 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.628767    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:22:59.629105 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.628814    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c" event={"ID":"50ad5e20-94c2-4467-a7e2-887364c10636","Type":"ContainerStarted","Data":"ea1c80d2731740788695a542ea9c22be207c9f4a90136870f740c498cf1042a2"}
Sep 10 19:22:59.630509 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.630485    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5" event={"ID":"82ddceb0-46f1-4ec7-bef6-a04726fd69cf","Type":"ContainerStarted","Data":"e0204039bc762429ae444fa3a06f99de9a997e1c7440eb105780d43f8813cb12"}
Sep 10 19:22:59.630594 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.630540    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:22:59.632230 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.632209    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6" event={"ID":"dfa86154-3d3c-43ed-acd0-cde0e365906b","Type":"ContainerStarted","Data":"c021042b01401cfa9fe867088230d296d73de2c8cd080829d4e710ff52a0c8e8"}
Sep 10 19:22:59.632342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.632255    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:22:59.632342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.632272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6" event={"ID":"dfa86154-3d3c-43ed-acd0-cde0e365906b","Type":"ContainerStarted","Data":"eb5b3744abc79d46db485cb2ace4af72947d5b851745abf00075e97782432835"}
Sep 10 19:22:59.656241 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.656197    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c" podStartSLOduration=1.656181973 podStartE2EDuration="1.656181973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:59.65404009 +0000 UTC m=+715.313837295" watchObservedRunningTime="2026-09-10 19:22:59.656181973 +0000 UTC m=+715.315979168"
Sep 10 19:22:59.728926 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.728888    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5" podStartSLOduration=2.085679863 podStartE2EDuration="2.728872108s" totalImagesPullingTime="643.192245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:59.726449795 +0000 UTC m=+715.386246987" watchObservedRunningTime="2026-09-10 19:22:59.728872108 +0000 UTC m=+715.388669306"
Sep 10 19:22:59.729540 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:22:59.729506    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6" podStartSLOduration=1.729498333 podStartE2EDuration="1.729498333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:59.690000379 +0000 UTC m=+715.349797573" watchObservedRunningTime="2026-09-10 19:22:59.729498333 +0000 UTC m=+715.389295585"
Sep 10 19:23:05.642937 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:05.642907    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:23:10.640790 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:10.640744    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:23:10.641208 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:10.640840    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:23:28.537165 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.537120    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"]
Sep 10 19:23:28.537755 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.537394    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5" podUID="82ddceb0-46f1-4ec7-bef6-a04726fd69cf" containerName="maas-api" containerID="cri-o://e0204039bc762429ae444fa3a06f99de9a997e1c7440eb105780d43f8813cb12" gracePeriod=30
Sep 10 19:23:28.601195 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.601162    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"]
Sep 10 19:23:28.601443 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.601415    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6" podUID="dfa86154-3d3c-43ed-acd0-cde0e365906b" containerName="payload-processing" containerID="cri-o://c021042b01401cfa9fe867088230d296d73de2c8cd080829d4e710ff52a0c8e8" gracePeriod=30
Sep 10 19:23:28.620775 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.620743    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"]
Sep 10 19:23:28.621016 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.620986    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c" podUID="50ad5e20-94c2-4467-a7e2-887364c10636" containerName="payload-pre-processing" containerID="cri-o://025f6143b0cc6ed17c3fe1e2ddfca67c550f52fbdde0ce41c97f947f42c78225" gracePeriod=30
Sep 10 19:23:28.800195 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.800142    2573 generic.go:356] "Generic (PLEG): container finished" podID="50ad5e20-94c2-4467-a7e2-887364c10636" containerID="025f6143b0cc6ed17c3fe1e2ddfca67c550f52fbdde0ce41c97f947f42c78225" exitCode=0
Sep 10 19:23:28.800378 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.800195    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c" event={"ID":"50ad5e20-94c2-4467-a7e2-887364c10636","Type":"ContainerDied","Data":"025f6143b0cc6ed17c3fe1e2ddfca67c550f52fbdde0ce41c97f947f42c78225"}
Sep 10 19:23:28.802255 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.802227    2573 generic.go:356] "Generic (PLEG): container finished" podID="82ddceb0-46f1-4ec7-bef6-a04726fd69cf" containerID="e0204039bc762429ae444fa3a06f99de9a997e1c7440eb105780d43f8813cb12" exitCode=0
Sep 10 19:23:28.802402 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.802308    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5" event={"ID":"82ddceb0-46f1-4ec7-bef6-a04726fd69cf","Type":"ContainerDied","Data":"e0204039bc762429ae444fa3a06f99de9a997e1c7440eb105780d43f8813cb12"}
Sep 10 19:23:28.804129 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.804109    2573 generic.go:356] "Generic (PLEG): container finished" podID="dfa86154-3d3c-43ed-acd0-cde0e365906b" containerID="c021042b01401cfa9fe867088230d296d73de2c8cd080829d4e710ff52a0c8e8" exitCode=0
Sep 10 19:23:28.804268 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.804218    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6" event={"ID":"dfa86154-3d3c-43ed-acd0-cde0e365906b","Type":"ContainerDied","Data":"c021042b01401cfa9fe867088230d296d73de2c8cd080829d4e710ff52a0c8e8"}
Sep 10 19:23:28.880810 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.880786    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:23:28.911542 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.911515    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:23:28.918504 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:28.918476    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:23:29.000571 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.000536    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx8rj\" (UniqueName: \"kubernetes.io/projected/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-kube-api-access-qx8rj\") pod \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") "
Sep 10 19:23:29.000571 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.000575    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-openshift-service-ca\") pod \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") "
Sep 10 19:23:29.000800 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.000597    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-maas-api-tls\") pod \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") "
Sep 10 19:23:29.000800 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.000621    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad5e20-94c2-4467-a7e2-887364c10636-plugins-config-volume\") pod \"50ad5e20-94c2-4467-a7e2-887364c10636\" (UID: \"50ad5e20-94c2-4467-a7e2-887364c10636\") "
Sep 10 19:23:29.000800 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.000675    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsc25\" (UniqueName: \"kubernetes.io/projected/50ad5e20-94c2-4467-a7e2-887364c10636-kube-api-access-zsc25\") pod \"50ad5e20-94c2-4467-a7e2-887364c10636\" (UID: \"50ad5e20-94c2-4467-a7e2-887364c10636\") "
Sep 10 19:23:29.000800 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.000727    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa86154-3d3c-43ed-acd0-cde0e365906b-plugins-config-volume\") pod \"dfa86154-3d3c-43ed-acd0-cde0e365906b\" (UID: \"dfa86154-3d3c-43ed-acd0-cde0e365906b\") "
Sep 10 19:23:29.000800 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.000771    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-888d5\" (UniqueName: \"kubernetes.io/projected/dfa86154-3d3c-43ed-acd0-cde0e365906b-kube-api-access-888d5\") pod \"dfa86154-3d3c-43ed-acd0-cde0e365906b\" (UID: \"dfa86154-3d3c-43ed-acd0-cde0e365906b\") "
Sep 10 19:23:29.001183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.000856    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-metrics-certs\") pod \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\" (UID: \"82ddceb0-46f1-4ec7-bef6-a04726fd69cf\") "
Sep 10 19:23:29.001260 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.001202    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82ddceb0-46f1-4ec7-bef6-a04726fd69cf" (UID: "82ddceb0-46f1-4ec7-bef6-a04726fd69cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:23:29.001260 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.001233    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa86154-3d3c-43ed-acd0-cde0e365906b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfa86154-3d3c-43ed-acd0-cde0e365906b" (UID: "dfa86154-3d3c-43ed-acd0-cde0e365906b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:23:29.001604 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.001580    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ad5e20-94c2-4467-a7e2-887364c10636-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50ad5e20-94c2-4467-a7e2-887364c10636" (UID: "50ad5e20-94c2-4467-a7e2-887364c10636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:23:29.003109 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.003075    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82ddceb0-46f1-4ec7-bef6-a04726fd69cf" (UID: "82ddceb0-46f1-4ec7-bef6-a04726fd69cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:23:29.003699 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.003668    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa86154-3d3c-43ed-acd0-cde0e365906b-kube-api-access-888d5" (OuterVolumeSpecName: "kube-api-access-888d5") pod "dfa86154-3d3c-43ed-acd0-cde0e365906b" (UID: "dfa86154-3d3c-43ed-acd0-cde0e365906b"). InnerVolumeSpecName "kube-api-access-888d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:23:29.003699 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.003684    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-kube-api-access-qx8rj" (OuterVolumeSpecName: "kube-api-access-qx8rj") pod "82ddceb0-46f1-4ec7-bef6-a04726fd69cf" (UID: "82ddceb0-46f1-4ec7-bef6-a04726fd69cf"). InnerVolumeSpecName "kube-api-access-qx8rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:23:29.003936 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.003918    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "82ddceb0-46f1-4ec7-bef6-a04726fd69cf" (UID: "82ddceb0-46f1-4ec7-bef6-a04726fd69cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:23:29.003983 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.003938    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ad5e20-94c2-4467-a7e2-887364c10636-kube-api-access-zsc25" (OuterVolumeSpecName: "kube-api-access-zsc25") pod "50ad5e20-94c2-4467-a7e2-887364c10636" (UID: "50ad5e20-94c2-4467-a7e2-887364c10636"). InnerVolumeSpecName "kube-api-access-zsc25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:23:29.102137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.102024    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8rj\" (UniqueName: \"kubernetes.io/projected/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-kube-api-access-qx8rj\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:23:29.102137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.102063    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:23:29.102137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.102076    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:23:29.102137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.102088    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ad5e20-94c2-4467-a7e2-887364c10636-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:23:29.102137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.102099    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsc25\" (UniqueName: \"kubernetes.io/projected/50ad5e20-94c2-4467-a7e2-887364c10636-kube-api-access-zsc25\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:23:29.102137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.102112    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa86154-3d3c-43ed-acd0-cde0e365906b-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:23:29.102137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.102124    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-888d5\" (UniqueName: \"kubernetes.io/projected/dfa86154-3d3c-43ed-acd0-cde0e365906b-kube-api-access-888d5\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:23:29.102137 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.102137    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82ddceb0-46f1-4ec7-bef6-a04726fd69cf-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:23:29.810592 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.810501    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"
Sep 10 19:23:29.810592 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.810518    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c" event={"ID":"50ad5e20-94c2-4467-a7e2-887364c10636","Type":"ContainerDied","Data":"ea1c80d2731740788695a542ea9c22be207c9f4a90136870f740c498cf1042a2"}
Sep 10 19:23:29.810592 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.810583    2573 scope.go:117] "RemoveContainer" containerID="025f6143b0cc6ed17c3fe1e2ddfca67c550f52fbdde0ce41c97f947f42c78225"
Sep 10 19:23:29.811959 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.811933    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5" event={"ID":"82ddceb0-46f1-4ec7-bef6-a04726fd69cf","Type":"ContainerDied","Data":"4bc1d7adbb7b3cd1b0eac000aa301ba439716db46e53c4da7bdb5246a6112f7c"}
Sep 10 19:23:29.812095 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.811941    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"
Sep 10 19:23:29.813531 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.813501    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"
Sep 10 19:23:29.813652 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.813549    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6" event={"ID":"dfa86154-3d3c-43ed-acd0-cde0e365906b","Type":"ContainerDied","Data":"eb5b3744abc79d46db485cb2ace4af72947d5b851745abf00075e97782432835"}
Sep 10 19:23:29.821814 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.821795    2573 scope.go:117] "RemoveContainer" containerID="e0204039bc762429ae444fa3a06f99de9a997e1c7440eb105780d43f8813cb12"
Sep 10 19:23:29.831867 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.831844    2573 scope.go:117] "RemoveContainer" containerID="c021042b01401cfa9fe867088230d296d73de2c8cd080829d4e710ff52a0c8e8"
Sep 10 19:23:29.854441 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.854407    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"]
Sep 10 19:23:29.862624 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.862598    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b59ba4a1-5dfd8f4686-2tbg6"]
Sep 10 19:23:29.883510 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.883474    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"]
Sep 10 19:23:29.892025 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.891996    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b59ba4a1-7f8bc5596d-p6ht5"]
Sep 10 19:23:29.912590 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.912557    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"]
Sep 10 19:23:29.919101 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:29.919075    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b59ba4a1-5b6547bb54-7v69c"]
Sep 10 19:23:30.833168 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:30.833124    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50ad5e20-94c2-4467-a7e2-887364c10636" path="/var/lib/kubelet/pods/50ad5e20-94c2-4467-a7e2-887364c10636/volumes"
Sep 10 19:23:30.833569 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:30.833553    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ddceb0-46f1-4ec7-bef6-a04726fd69cf" path="/var/lib/kubelet/pods/82ddceb0-46f1-4ec7-bef6-a04726fd69cf/volumes"
Sep 10 19:23:30.833859 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:30.833848    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa86154-3d3c-43ed-acd0-cde0e365906b" path="/var/lib/kubelet/pods/dfa86154-3d3c-43ed-acd0-cde0e365906b/volumes"
Sep 10 19:23:33.403302 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403270    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"]
Sep 10 19:23:33.403783 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403681    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50ad5e20-94c2-4467-a7e2-887364c10636" containerName="payload-pre-processing"
Sep 10 19:23:33.403783 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403692    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="50ad5e20-94c2-4467-a7e2-887364c10636" containerName="payload-pre-processing"
Sep 10 19:23:33.403783 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403709    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa86154-3d3c-43ed-acd0-cde0e365906b" containerName="payload-processing"
Sep 10 19:23:33.403783 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403714    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa86154-3d3c-43ed-acd0-cde0e365906b" containerName="payload-processing"
Sep 10 19:23:33.403783 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403740    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ddceb0-46f1-4ec7-bef6-a04726fd69cf" containerName="maas-api"
Sep 10 19:23:33.403783 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403746    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ddceb0-46f1-4ec7-bef6-a04726fd69cf" containerName="maas-api"
Sep 10 19:23:33.404095 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403805    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="50ad5e20-94c2-4467-a7e2-887364c10636" containerName="payload-pre-processing"
Sep 10 19:23:33.404095 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403814    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa86154-3d3c-43ed-acd0-cde0e365906b" containerName="payload-processing"
Sep 10 19:23:33.404095 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.403820    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ddceb0-46f1-4ec7-bef6-a04726fd69cf" containerName="maas-api"
Sep 10 19:23:33.408741 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.408717    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.411448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.411422    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f21c165c-serving-cert\""
Sep 10 19:23:33.411625 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.411604    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 19:23:33.411702 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.411609    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lw7bv\""
Sep 10 19:23:33.420606 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.420575    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"]
Sep 10 19:23:33.546817 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.546779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-maas-api-tls\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.546817 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.546815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4aad05-3bab-41f5-8555-f3da52156db2-openshift-service-ca\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.547036 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.546843    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trb88\" (UniqueName: \"kubernetes.io/projected/be4aad05-3bab-41f5-8555-f3da52156db2-kube-api-access-trb88\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.547036 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.546934    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-metrics-certs\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.648334 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.648288    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-maas-api-tls\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.648334 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.648330    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4aad05-3bab-41f5-8555-f3da52156db2-openshift-service-ca\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.648546 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.648355    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trb88\" (UniqueName: \"kubernetes.io/projected/be4aad05-3bab-41f5-8555-f3da52156db2-kube-api-access-trb88\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.648546 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.648401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-metrics-certs\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.648546 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:23:33.648453    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-serving-cert: secret "maas-api-e2e-ait-f21c165c-serving-cert" not found
Sep 10 19:23:33.648546 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:23:33.648535    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-maas-api-tls podName:be4aad05-3bab-41f5-8555-f3da52156db2 nodeName:}" failed. No retries permitted until 2026-09-10 19:23:34.148518128 +0000 UTC m=+749.808315331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-maas-api-tls") pod "maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl" (UID: "be4aad05-3bab-41f5-8555-f3da52156db2") : secret "maas-api-e2e-ait-f21c165c-serving-cert" not found
Sep 10 19:23:33.649066 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.649048    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4aad05-3bab-41f5-8555-f3da52156db2-openshift-service-ca\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.650889 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.650872    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-metrics-certs\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:33.661859 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:33.661796    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trb88\" (UniqueName: \"kubernetes.io/projected/be4aad05-3bab-41f5-8555-f3da52156db2-kube-api-access-trb88\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:34.152623 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.152584    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-maas-api-tls\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:34.155254 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.155220    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-maas-api-tls\") pod \"maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:34.320776 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.320737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:34.542791 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.542751    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"]
Sep 10 19:23:34.546675 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.546653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:34.550326 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.550295    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f21c165c\""
Sep 10 19:23:34.550326 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.550302    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f21c165c-dockercfg-wt5mg\""
Sep 10 19:23:34.558811 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.558784    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"]
Sep 10 19:23:34.620594 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.620573    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"]
Sep 10 19:23:34.622031 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:23:34.622000    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4aad05_3bab_41f5_8555_f3da52156db2.slice/crio-10d0517c3b503d41711edd79fec016463fde75b014923f0c325e204d38183b94 WatchSource:0}: Error finding container 10d0517c3b503d41711edd79fec016463fde75b014923f0c325e204d38183b94: Status 404 returned error can't find the container with id 10d0517c3b503d41711edd79fec016463fde75b014923f0c325e204d38183b94
Sep 10 19:23:34.649953 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.649921    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"]
Sep 10 19:23:34.658305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.658273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c0d7a7-fa51-47d5-b04f-621b28888c47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl\" (UID: \"05c0d7a7-fa51-47d5-b04f-621b28888c47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:34.658443 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.658368    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smr65\" (UniqueName: \"kubernetes.io/projected/05c0d7a7-fa51-47d5-b04f-621b28888c47-kube-api-access-smr65\") pod \"payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl\" (UID: \"05c0d7a7-fa51-47d5-b04f-621b28888c47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:34.658688 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.658670    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:34.668258 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.668218    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"]
Sep 10 19:23:34.759653 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.759618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c0d7a7-fa51-47d5-b04f-621b28888c47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl\" (UID: \"05c0d7a7-fa51-47d5-b04f-621b28888c47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:34.759880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.759703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smr65\" (UniqueName: \"kubernetes.io/projected/05c0d7a7-fa51-47d5-b04f-621b28888c47-kube-api-access-smr65\") pod \"payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl\" (UID: \"05c0d7a7-fa51-47d5-b04f-621b28888c47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:34.759880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.759769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-plugins-config-volume\") pod \"payload-processing-e2e-ait-f21c165c-d9587d877-46xd4\" (UID: \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\") " pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:34.759880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.759806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9xh\" (UniqueName: \"kubernetes.io/projected/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-kube-api-access-sz9xh\") pod \"payload-processing-e2e-ait-f21c165c-d9587d877-46xd4\" (UID: \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\") " pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:34.760341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.760320    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c0d7a7-fa51-47d5-b04f-621b28888c47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl\" (UID: \"05c0d7a7-fa51-47d5-b04f-621b28888c47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:34.768795 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.768763    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smr65\" (UniqueName: \"kubernetes.io/projected/05c0d7a7-fa51-47d5-b04f-621b28888c47-kube-api-access-smr65\") pod \"payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl\" (UID: \"05c0d7a7-fa51-47d5-b04f-621b28888c47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:34.845660 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.845570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl" event={"ID":"be4aad05-3bab-41f5-8555-f3da52156db2","Type":"ContainerStarted","Data":"10d0517c3b503d41711edd79fec016463fde75b014923f0c325e204d38183b94"}
Sep 10 19:23:34.858794 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.858752    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:34.861205 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.861180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-plugins-config-volume\") pod \"payload-processing-e2e-ait-f21c165c-d9587d877-46xd4\" (UID: \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\") " pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:34.861275 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.861228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9xh\" (UniqueName: \"kubernetes.io/projected/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-kube-api-access-sz9xh\") pod \"payload-processing-e2e-ait-f21c165c-d9587d877-46xd4\" (UID: \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\") " pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:34.861968 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.861948    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-plugins-config-volume\") pod \"payload-processing-e2e-ait-f21c165c-d9587d877-46xd4\" (UID: \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\") " pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:34.870443 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.870415    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9xh\" (UniqueName: \"kubernetes.io/projected/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-kube-api-access-sz9xh\") pod \"payload-processing-e2e-ait-f21c165c-d9587d877-46xd4\" (UID: \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\") " pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:34.970585 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:34.970552    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:35.140026 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.140000    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"]
Sep 10 19:23:35.142124 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:23:35.142099    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c0d7a7_fa51_47d5_b04f_621b28888c47.slice/crio-c419a8d6a90100e7a652e1ec6a040d49f3d16c25fde44d5efcf6e76b38726259 WatchSource:0}: Error finding container c419a8d6a90100e7a652e1ec6a040d49f3d16c25fde44d5efcf6e76b38726259: Status 404 returned error can't find the container with id c419a8d6a90100e7a652e1ec6a040d49f3d16c25fde44d5efcf6e76b38726259
Sep 10 19:23:35.255407 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.254966    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"]
Sep 10 19:23:35.853516 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.853481    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4" event={"ID":"dd30bded-c74e-4fa8-91e4-8b8d71f9e166","Type":"ContainerStarted","Data":"4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108"}
Sep 10 19:23:35.854002 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.853529    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4" event={"ID":"dd30bded-c74e-4fa8-91e4-8b8d71f9e166","Type":"ContainerStarted","Data":"a63357fdc0fb58bab25d48dff5e20e19c0a118630cfff6378e5e9331d22c2888"}
Sep 10 19:23:35.854002 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.853576    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:35.855174 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.855126    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl" event={"ID":"be4aad05-3bab-41f5-8555-f3da52156db2","Type":"ContainerStarted","Data":"56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451"}
Sep 10 19:23:35.855310 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.855277    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:35.856803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.856780    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl" event={"ID":"05c0d7a7-fa51-47d5-b04f-621b28888c47","Type":"ContainerStarted","Data":"552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e"}
Sep 10 19:23:35.856880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.856825    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl" event={"ID":"05c0d7a7-fa51-47d5-b04f-621b28888c47","Type":"ContainerStarted","Data":"c419a8d6a90100e7a652e1ec6a040d49f3d16c25fde44d5efcf6e76b38726259"}
Sep 10 19:23:35.856880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.856852    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:35.878135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.878081    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4" podStartSLOduration=1.87806234 podStartE2EDuration="1.87806234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:23:35.875894139 +0000 UTC m=+751.535691334" watchObservedRunningTime="2026-09-10 19:23:35.87806234 +0000 UTC m=+751.537859534"
Sep 10 19:23:35.916778 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.916713    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl" podStartSLOduration=2.478169776 podStartE2EDuration="2.916690691s" totalImagesPullingTime="438.520915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:23:35.916142058 +0000 UTC m=+751.575939264" watchObservedRunningTime="2026-09-10 19:23:35.916690691 +0000 UTC m=+751.576487897"
Sep 10 19:23:35.956074 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:35.956014    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl" podStartSLOduration=1.955992378 podStartE2EDuration="1.955992378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:23:35.954362531 +0000 UTC m=+751.614159752" watchObservedRunningTime="2026-09-10 19:23:35.955992378 +0000 UTC m=+751.615789575"
Sep 10 19:23:39.211594 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.211555    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"]
Sep 10 19:23:39.215470 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.215448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.218832 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.218797    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3c7a5e\"/\"kube-root-ca.crt\""
Sep 10 19:23:39.219704 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.219679    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3c7a5e\"/\"autores-model-3c7a5e-kserve-self-signed-certs\""
Sep 10 19:23:39.220007 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.219991    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-3c7a5e\"/\"openshift-service-ca.crt\""
Sep 10 19:23:39.220097 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.220019    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-3c7a5e\"/\"default-dockercfg-nvm5t\""
Sep 10 19:23:39.228434 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.228406    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"]
Sep 10 19:23:39.307294 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.307243    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-home\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.307294 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.307298    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnnm4\" (UniqueName: \"kubernetes.io/projected/2da117fc-89e5-42a6-aaab-a3294943e5e3-kube-api-access-qnnm4\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.307549 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.307426    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-model-cache\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.307549 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.307455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-tmp-dir\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.307549 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.307485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-dshm\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.307688 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.307552    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da117fc-89e5-42a6-aaab-a3294943e5e3-tls-certs\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.408110 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.408064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-home\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.408110 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.408110    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnnm4\" (UniqueName: \"kubernetes.io/projected/2da117fc-89e5-42a6-aaab-a3294943e5e3-kube-api-access-qnnm4\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.408341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.408213    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-model-cache\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.408341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.408240    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-tmp-dir\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.408341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.408275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-dshm\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.408341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.408321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da117fc-89e5-42a6-aaab-a3294943e5e3-tls-certs\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.408566 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.408540    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-home\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.408634 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.408607    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-tmp-dir\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.408679 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.408649    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-model-cache\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.410702 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.410672    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-dshm\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.410958 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.410940    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da117fc-89e5-42a6-aaab-a3294943e5e3-tls-certs\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.421988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.421957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnnm4\" (UniqueName: \"kubernetes.io/projected/2da117fc-89e5-42a6-aaab-a3294943e5e3-kube-api-access-qnnm4\") pod \"autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") " pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.526234 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.526193    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:39.807483 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.807450    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"]
Sep 10 19:23:39.809364 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:23:39.809339    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2da117fc_89e5_42a6_aaab_a3294943e5e3.slice/crio-418518a972563174e295a473b6e92ecadcd80179bf84bc69206cad2b7aed5c71 WatchSource:0}: Error finding container 418518a972563174e295a473b6e92ecadcd80179bf84bc69206cad2b7aed5c71: Status 404 returned error can't find the container with id 418518a972563174e295a473b6e92ecadcd80179bf84bc69206cad2b7aed5c71
Sep 10 19:23:39.880627 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.880591    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf" event={"ID":"2da117fc-89e5-42a6-aaab-a3294943e5e3","Type":"ContainerStarted","Data":"83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2"}
Sep 10 19:23:39.880811 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.880645    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf" event={"ID":"2da117fc-89e5-42a6-aaab-a3294943e5e3","Type":"ContainerStarted","Data":"418518a972563174e295a473b6e92ecadcd80179bf84bc69206cad2b7aed5c71"}
Sep 10 19:23:39.904863 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:39.904756    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf" podStartSLOduration=0.904740075 podStartE2EDuration="904.740075ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:23:39.9038388 +0000 UTC m=+755.563635994" watchObservedRunningTime="2026-09-10 19:23:39.904740075 +0000 UTC m=+755.564537269"
Sep 10 19:23:41.868125 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:41.868098    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:23:46.864370 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:46.864333    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:23:46.864801 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:46.864618    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:23:49.526543 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:49.526501    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:49.526986 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:49.526555    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:49.539418 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:49.539387    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:23:49.947689 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:23:49.947598    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:24:04.018870 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.018831    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"]
Sep 10 19:24:04.019464 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.019103    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl" podUID="be4aad05-3bab-41f5-8555-f3da52156db2" containerName="maas-api" containerID="cri-o://56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451" gracePeriod=30
Sep 10 19:24:04.072357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.072321    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"]
Sep 10 19:24:04.072600 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.072571    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4" podUID="dd30bded-c74e-4fa8-91e4-8b8d71f9e166" containerName="payload-processing" containerID="cri-o://4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108" gracePeriod=30
Sep 10 19:24:04.077635 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.077605    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"]
Sep 10 19:24:04.077883 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.077846    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl" podUID="05c0d7a7-fa51-47d5-b04f-621b28888c47" containerName="payload-pre-processing" containerID="cri-o://552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e" gracePeriod=30
Sep 10 19:24:04.347689 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.347661    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:24:04.364005 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.363977    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:24:04.368995 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.368971    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:24:04.459403 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.459366    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz9xh\" (UniqueName: \"kubernetes.io/projected/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-kube-api-access-sz9xh\") pod \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\" (UID: \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\") "
Sep 10 19:24:04.459574 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.459414    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-metrics-certs\") pod \"be4aad05-3bab-41f5-8555-f3da52156db2\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") "
Sep 10 19:24:04.459574 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.459451    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smr65\" (UniqueName: \"kubernetes.io/projected/05c0d7a7-fa51-47d5-b04f-621b28888c47-kube-api-access-smr65\") pod \"05c0d7a7-fa51-47d5-b04f-621b28888c47\" (UID: \"05c0d7a7-fa51-47d5-b04f-621b28888c47\") "
Sep 10 19:24:04.459574 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.459495    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c0d7a7-fa51-47d5-b04f-621b28888c47-plugins-config-volume\") pod \"05c0d7a7-fa51-47d5-b04f-621b28888c47\" (UID: \"05c0d7a7-fa51-47d5-b04f-621b28888c47\") "
Sep 10 19:24:04.459574 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.459526    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-maas-api-tls\") pod \"be4aad05-3bab-41f5-8555-f3da52156db2\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") "
Sep 10 19:24:04.459830 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.459687    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4aad05-3bab-41f5-8555-f3da52156db2-openshift-service-ca\") pod \"be4aad05-3bab-41f5-8555-f3da52156db2\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") "
Sep 10 19:24:04.459830 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.459742    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trb88\" (UniqueName: \"kubernetes.io/projected/be4aad05-3bab-41f5-8555-f3da52156db2-kube-api-access-trb88\") pod \"be4aad05-3bab-41f5-8555-f3da52156db2\" (UID: \"be4aad05-3bab-41f5-8555-f3da52156db2\") "
Sep 10 19:24:04.459830 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.459768    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-plugins-config-volume\") pod \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\" (UID: \"dd30bded-c74e-4fa8-91e4-8b8d71f9e166\") "
Sep 10 19:24:04.460013 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.459962    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c0d7a7-fa51-47d5-b04f-621b28888c47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05c0d7a7-fa51-47d5-b04f-621b28888c47" (UID: "05c0d7a7-fa51-47d5-b04f-621b28888c47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:24:04.460255 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.460222    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c0d7a7-fa51-47d5-b04f-621b28888c47-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:04.460385 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.460271    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4aad05-3bab-41f5-8555-f3da52156db2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be4aad05-3bab-41f5-8555-f3da52156db2" (UID: "be4aad05-3bab-41f5-8555-f3da52156db2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:24:04.460385 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.460332    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd30bded-c74e-4fa8-91e4-8b8d71f9e166" (UID: "dd30bded-c74e-4fa8-91e4-8b8d71f9e166"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:24:04.461778 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.461757    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "be4aad05-3bab-41f5-8555-f3da52156db2" (UID: "be4aad05-3bab-41f5-8555-f3da52156db2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:24:04.462317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.462291    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be4aad05-3bab-41f5-8555-f3da52156db2" (UID: "be4aad05-3bab-41f5-8555-f3da52156db2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:24:04.462317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.462298    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-kube-api-access-sz9xh" (OuterVolumeSpecName: "kube-api-access-sz9xh") pod "dd30bded-c74e-4fa8-91e4-8b8d71f9e166" (UID: "dd30bded-c74e-4fa8-91e4-8b8d71f9e166"). InnerVolumeSpecName "kube-api-access-sz9xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:24:04.462493 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.462473    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4aad05-3bab-41f5-8555-f3da52156db2-kube-api-access-trb88" (OuterVolumeSpecName: "kube-api-access-trb88") pod "be4aad05-3bab-41f5-8555-f3da52156db2" (UID: "be4aad05-3bab-41f5-8555-f3da52156db2"). InnerVolumeSpecName "kube-api-access-trb88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:24:04.462532 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.462490    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c0d7a7-fa51-47d5-b04f-621b28888c47-kube-api-access-smr65" (OuterVolumeSpecName: "kube-api-access-smr65") pod "05c0d7a7-fa51-47d5-b04f-621b28888c47" (UID: "05c0d7a7-fa51-47d5-b04f-621b28888c47"). InnerVolumeSpecName "kube-api-access-smr65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:24:04.561420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.561331    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz9xh\" (UniqueName: \"kubernetes.io/projected/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-kube-api-access-sz9xh\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:04.561420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.561364    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:04.561420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.561373    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smr65\" (UniqueName: \"kubernetes.io/projected/05c0d7a7-fa51-47d5-b04f-621b28888c47-kube-api-access-smr65\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:04.561420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.561384    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4aad05-3bab-41f5-8555-f3da52156db2-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:04.561420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.561392    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4aad05-3bab-41f5-8555-f3da52156db2-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:04.561420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.561400    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trb88\" (UniqueName: \"kubernetes.io/projected/be4aad05-3bab-41f5-8555-f3da52156db2-kube-api-access-trb88\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:04.561420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:04.561409    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd30bded-c74e-4fa8-91e4-8b8d71f9e166-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:05.019269 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.019229    2573 generic.go:356] "Generic (PLEG): container finished" podID="dd30bded-c74e-4fa8-91e4-8b8d71f9e166" containerID="4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108" exitCode=0
Sep 10 19:24:05.019770 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.019298    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"
Sep 10 19:24:05.019770 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.019311    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4" event={"ID":"dd30bded-c74e-4fa8-91e4-8b8d71f9e166","Type":"ContainerDied","Data":"4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108"}
Sep 10 19:24:05.019770 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.019372    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4" event={"ID":"dd30bded-c74e-4fa8-91e4-8b8d71f9e166","Type":"ContainerDied","Data":"a63357fdc0fb58bab25d48dff5e20e19c0a118630cfff6378e5e9331d22c2888"}
Sep 10 19:24:05.019770 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.019390    2573 scope.go:117] "RemoveContainer" containerID="4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108"
Sep 10 19:24:05.021074 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.021049    2573 generic.go:356] "Generic (PLEG): container finished" podID="be4aad05-3bab-41f5-8555-f3da52156db2" containerID="56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451" exitCode=0
Sep 10 19:24:05.021245 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.021075    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl" event={"ID":"be4aad05-3bab-41f5-8555-f3da52156db2","Type":"ContainerDied","Data":"56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451"}
Sep 10 19:24:05.021245 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.021125    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"
Sep 10 19:24:05.021245 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.021174    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl" event={"ID":"be4aad05-3bab-41f5-8555-f3da52156db2","Type":"ContainerDied","Data":"10d0517c3b503d41711edd79fec016463fde75b014923f0c325e204d38183b94"}
Sep 10 19:24:05.022762 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.022728    2573 generic.go:356] "Generic (PLEG): container finished" podID="05c0d7a7-fa51-47d5-b04f-621b28888c47" containerID="552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e" exitCode=0
Sep 10 19:24:05.022762 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.022750    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl" event={"ID":"05c0d7a7-fa51-47d5-b04f-621b28888c47","Type":"ContainerDied","Data":"552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e"}
Sep 10 19:24:05.022961 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.022778    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"
Sep 10 19:24:05.022961 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.022789    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl" event={"ID":"05c0d7a7-fa51-47d5-b04f-621b28888c47","Type":"ContainerDied","Data":"c419a8d6a90100e7a652e1ec6a040d49f3d16c25fde44d5efcf6e76b38726259"}
Sep 10 19:24:05.030355 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.030336    2573 scope.go:117] "RemoveContainer" containerID="4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108"
Sep 10 19:24:05.030639 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:24:05.030618    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108\": container with ID starting with 4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108 not found: ID does not exist" containerID="4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108"
Sep 10 19:24:05.030736 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.030650    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108"} err="failed to get container status \"4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108\": rpc error: code = NotFound desc = could not find container \"4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108\": container with ID starting with 4757d4782e5e0545e675979a0969e45b3c6ebc5055734da9dde93d25a4479108 not found: ID does not exist"
Sep 10 19:24:05.030736 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.030674    2573 scope.go:117] "RemoveContainer" containerID="56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451"
Sep 10 19:24:05.046049 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.046017    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"]
Sep 10 19:24:05.048007 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.047984    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f21c165c-d9587d877-46xd4"]
Sep 10 19:24:05.048792 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.048775    2573 scope.go:117] "RemoveContainer" containerID="56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451"
Sep 10 19:24:05.049172 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:24:05.049129    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451\": container with ID starting with 56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451 not found: ID does not exist" containerID="56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451"
Sep 10 19:24:05.049238 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.049178    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451"} err="failed to get container status \"56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451\": rpc error: code = NotFound desc = could not find container \"56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451\": container with ID starting with 56532d07b9099bbecb1edb17b331ba2863a57680dc47cf3567892ea0c086e451 not found: ID does not exist"
Sep 10 19:24:05.049238 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.049201    2573 scope.go:117] "RemoveContainer" containerID="552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e"
Sep 10 19:24:05.057923 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.057903    2573 scope.go:117] "RemoveContainer" containerID="552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e"
Sep 10 19:24:05.058321 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:24:05.058301    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e\": container with ID starting with 552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e not found: ID does not exist" containerID="552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e"
Sep 10 19:24:05.058409 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.058327    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e"} err="failed to get container status \"552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e\": rpc error: code = NotFound desc = could not find container \"552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e\": container with ID starting with 552ecea25254d23325ba5cfbb1e2ec57fa5b87321964ac92a5e266153083cf8e not found: ID does not exist"
Sep 10 19:24:05.059582 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.059563    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"]
Sep 10 19:24:05.065356 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.065329    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f21c165c-7d4cbc55c5-8qwtl"]
Sep 10 19:24:05.077880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.077846    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"]
Sep 10 19:24:05.083484 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:05.083449    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f21c165c-55bf9dbd78-98qgl"]
Sep 10 19:24:06.833677 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:06.833636    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05c0d7a7-fa51-47d5-b04f-621b28888c47" path="/var/lib/kubelet/pods/05c0d7a7-fa51-47d5-b04f-621b28888c47/volumes"
Sep 10 19:24:06.834191 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:06.834031    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be4aad05-3bab-41f5-8555-f3da52156db2" path="/var/lib/kubelet/pods/be4aad05-3bab-41f5-8555-f3da52156db2/volumes"
Sep 10 19:24:06.834374 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:06.834360    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd30bded-c74e-4fa8-91e4-8b8d71f9e166" path="/var/lib/kubelet/pods/dd30bded-c74e-4fa8-91e4-8b8d71f9e166/volumes"
Sep 10 19:24:09.531141 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531105    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"]
Sep 10 19:24:09.531618 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531603    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4aad05-3bab-41f5-8555-f3da52156db2" containerName="maas-api"
Sep 10 19:24:09.531618 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531619    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4aad05-3bab-41f5-8555-f3da52156db2" containerName="maas-api"
Sep 10 19:24:09.531701 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531630    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c0d7a7-fa51-47d5-b04f-621b28888c47" containerName="payload-pre-processing"
Sep 10 19:24:09.531701 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531640    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c0d7a7-fa51-47d5-b04f-621b28888c47" containerName="payload-pre-processing"
Sep 10 19:24:09.531701 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531650    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd30bded-c74e-4fa8-91e4-8b8d71f9e166" containerName="payload-processing"
Sep 10 19:24:09.531701 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531657    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd30bded-c74e-4fa8-91e4-8b8d71f9e166" containerName="payload-processing"
Sep 10 19:24:09.531850 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531732    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c0d7a7-fa51-47d5-b04f-621b28888c47" containerName="payload-pre-processing"
Sep 10 19:24:09.531850 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531752    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="be4aad05-3bab-41f5-8555-f3da52156db2" containerName="maas-api"
Sep 10 19:24:09.531850 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.531760    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd30bded-c74e-4fa8-91e4-8b8d71f9e166" containerName="payload-processing"
Sep 10 19:24:09.536496 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.536479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.539495 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.539470    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a888f7aa-serving-cert\""
Sep 10 19:24:09.539870 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.539855    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lw7bv\""
Sep 10 19:24:09.540667 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.540652    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 19:24:09.552336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.552315    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"]
Sep 10 19:24:09.607627 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.607593    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-openshift-service-ca\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.607788 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.607641    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-metrics-certs\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.607788 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.607695    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-maas-api-tls\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.607924 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.607826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fgxg\" (UniqueName: \"kubernetes.io/projected/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-kube-api-access-2fgxg\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.709239 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.709192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-maas-api-tls\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.709446 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.709299    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgxg\" (UniqueName: \"kubernetes.io/projected/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-kube-api-access-2fgxg\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.709446 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.709364    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-openshift-service-ca\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.709446 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.709410    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-metrics-certs\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.709911 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.709889    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-openshift-service-ca\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.711809 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.711788    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-metrics-certs\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.711903 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.711843    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-maas-api-tls\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.724458 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.724421    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgxg\" (UniqueName: \"kubernetes.io/projected/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-kube-api-access-2fgxg\") pod \"maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:09.847044 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:09.846947    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:10.127914 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.127832    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"]
Sep 10 19:24:10.131649 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:24:10.131622    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef02976e_1c47_4ab4_b4fb_f5b6e6cd6475.slice/crio-2c161f3331f9c776187ff1e1bb47d84fc966db052d75aeaacc86b2365ca030ff WatchSource:0}: Error finding container 2c161f3331f9c776187ff1e1bb47d84fc966db052d75aeaacc86b2365ca030ff: Status 404 returned error can't find the container with id 2c161f3331f9c776187ff1e1bb47d84fc966db052d75aeaacc86b2365ca030ff
Sep 10 19:24:10.596337 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.596303    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"]
Sep 10 19:24:10.600193 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.600139    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:10.602986 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.602963    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a888f7aa-dockercfg-xd6dj\""
Sep 10 19:24:10.603396 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.603269    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a888f7aa\""
Sep 10 19:24:10.614576 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.614548    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"]
Sep 10 19:24:10.660979 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.660938    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"]
Sep 10 19:24:10.666477 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.666447    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:10.683729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.683699    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"]
Sep 10 19:24:10.720633 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.720594    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd01808-3601-4ebf-a121-45dd0f79d8e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg\" (UID: \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:10.720869 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.720678    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ts2\" (UniqueName: \"kubernetes.io/projected/3bd01808-3601-4ebf-a121-45dd0f79d8e0-kube-api-access-v8ts2\") pod \"payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg\" (UID: \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:10.821710 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.821663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpjlb\" (UniqueName: \"kubernetes.io/projected/b9e878ec-79f1-44c9-84ea-def49999ec85-kube-api-access-hpjlb\") pod \"payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq\" (UID: \"b9e878ec-79f1-44c9-84ea-def49999ec85\") " pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:10.821710 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.821718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ts2\" (UniqueName: \"kubernetes.io/projected/3bd01808-3601-4ebf-a121-45dd0f79d8e0-kube-api-access-v8ts2\") pod \"payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg\" (UID: \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:10.821949 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.821805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e878ec-79f1-44c9-84ea-def49999ec85-plugins-config-volume\") pod \"payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq\" (UID: \"b9e878ec-79f1-44c9-84ea-def49999ec85\") " pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:10.822005 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.821983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd01808-3601-4ebf-a121-45dd0f79d8e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg\" (UID: \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:10.822728 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.822702    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd01808-3601-4ebf-a121-45dd0f79d8e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg\" (UID: \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:10.831542 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.831514    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ts2\" (UniqueName: \"kubernetes.io/projected/3bd01808-3601-4ebf-a121-45dd0f79d8e0-kube-api-access-v8ts2\") pod \"payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg\" (UID: \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:10.911977 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.911937    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:10.923646 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.923612    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpjlb\" (UniqueName: \"kubernetes.io/projected/b9e878ec-79f1-44c9-84ea-def49999ec85-kube-api-access-hpjlb\") pod \"payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq\" (UID: \"b9e878ec-79f1-44c9-84ea-def49999ec85\") " pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:10.923802 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.923714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e878ec-79f1-44c9-84ea-def49999ec85-plugins-config-volume\") pod \"payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq\" (UID: \"b9e878ec-79f1-44c9-84ea-def49999ec85\") " pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:10.924477 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.924456    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e878ec-79f1-44c9-84ea-def49999ec85-plugins-config-volume\") pod \"payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq\" (UID: \"b9e878ec-79f1-44c9-84ea-def49999ec85\") " pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:10.933401 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.933373    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpjlb\" (UniqueName: \"kubernetes.io/projected/b9e878ec-79f1-44c9-84ea-def49999ec85-kube-api-access-hpjlb\") pod \"payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq\" (UID: \"b9e878ec-79f1-44c9-84ea-def49999ec85\") " pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:10.989276 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:10.989234    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:11.062425 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:11.062378    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn" event={"ID":"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475","Type":"ContainerStarted","Data":"1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531"}
Sep 10 19:24:11.062643 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:11.062452    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn" event={"ID":"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475","Type":"ContainerStarted","Data":"2c161f3331f9c776187ff1e1bb47d84fc966db052d75aeaacc86b2365ca030ff"}
Sep 10 19:24:11.062643 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:11.062608    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:11.101284 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:11.101162    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn" podStartSLOduration=1.753803745 podStartE2EDuration="2.101127015s" totalImagesPullingTime="347.32327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:24:11.099265797 +0000 UTC m=+786.759062990" watchObservedRunningTime="2026-09-10 19:24:11.101127015 +0000 UTC m=+786.760924220"
Sep 10 19:24:11.196351 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:11.196323    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"]
Sep 10 19:24:11.198544 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:24:11.198515    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd01808_3601_4ebf_a121_45dd0f79d8e0.slice/crio-71943b4ad6acc2ff750b4f4f89e5e1f3b8f58d4e592f3a2c9db72fb147d3aa41 WatchSource:0}: Error finding container 71943b4ad6acc2ff750b4f4f89e5e1f3b8f58d4e592f3a2c9db72fb147d3aa41: Status 404 returned error can't find the container with id 71943b4ad6acc2ff750b4f4f89e5e1f3b8f58d4e592f3a2c9db72fb147d3aa41
Sep 10 19:24:11.273507 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:11.273480    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"]
Sep 10 19:24:11.275484 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:24:11.275454    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e878ec_79f1_44c9_84ea_def49999ec85.slice/crio-149d2643c91645dabe3e9b9f00beaa99b52dd4c012ecb7ce5f705997f4e356f3 WatchSource:0}: Error finding container 149d2643c91645dabe3e9b9f00beaa99b52dd4c012ecb7ce5f705997f4e356f3: Status 404 returned error can't find the container with id 149d2643c91645dabe3e9b9f00beaa99b52dd4c012ecb7ce5f705997f4e356f3
Sep 10 19:24:12.069544 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:12.069506    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq" event={"ID":"b9e878ec-79f1-44c9-84ea-def49999ec85","Type":"ContainerStarted","Data":"070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea"}
Sep 10 19:24:12.070047 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:12.069577    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq" event={"ID":"b9e878ec-79f1-44c9-84ea-def49999ec85","Type":"ContainerStarted","Data":"149d2643c91645dabe3e9b9f00beaa99b52dd4c012ecb7ce5f705997f4e356f3"}
Sep 10 19:24:12.070047 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:12.069607    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:12.071218 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:12.071194    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg" event={"ID":"3bd01808-3601-4ebf-a121-45dd0f79d8e0","Type":"ContainerStarted","Data":"bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa"}
Sep 10 19:24:12.071285 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:12.071242    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg" event={"ID":"3bd01808-3601-4ebf-a121-45dd0f79d8e0","Type":"ContainerStarted","Data":"71943b4ad6acc2ff750b4f4f89e5e1f3b8f58d4e592f3a2c9db72fb147d3aa41"}
Sep 10 19:24:12.071407 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:12.071394    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:12.092712 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:12.092661    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq" podStartSLOduration=2.092642704 podStartE2EDuration="2.092642704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:24:12.090694365 +0000 UTC m=+787.750491559" watchObservedRunningTime="2026-09-10 19:24:12.092642704 +0000 UTC m=+787.752439907"
Sep 10 19:24:12.109548 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:12.109487    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg" podStartSLOduration=2.109462116 podStartE2EDuration="2.109462116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:24:12.108870784 +0000 UTC m=+787.768667969" watchObservedRunningTime="2026-09-10 19:24:12.109462116 +0000 UTC m=+787.769259310"
Sep 10 19:24:17.076903 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:17.076874    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:21.961912 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:21.961880    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"]
Sep 10 19:24:21.962464 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:21.962090    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j" podUID="3278c2e2-7b45-41aa-bd26-941c69ff893b" containerName="payload-processing" containerID="cri-o://63d8ae8bf9e0f663b02b60d238d71152123d015f493ffd598e166db16fbb0c6a" gracePeriod=30
Sep 10 19:24:22.130113 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:22.130061    2573 generic.go:356] "Generic (PLEG): container finished" podID="3278c2e2-7b45-41aa-bd26-941c69ff893b" containerID="63d8ae8bf9e0f663b02b60d238d71152123d015f493ffd598e166db16fbb0c6a" exitCode=0
Sep 10 19:24:22.130273 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:22.130134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j" event={"ID":"3278c2e2-7b45-41aa-bd26-941c69ff893b","Type":"ContainerDied","Data":"63d8ae8bf9e0f663b02b60d238d71152123d015f493ffd598e166db16fbb0c6a"}
Sep 10 19:24:22.229002 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:22.228934    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:24:22.339552 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:22.339512    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278c2e2-7b45-41aa-bd26-941c69ff893b-plugins-config-volume\") pod \"3278c2e2-7b45-41aa-bd26-941c69ff893b\" (UID: \"3278c2e2-7b45-41aa-bd26-941c69ff893b\") "
Sep 10 19:24:22.339552 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:22.339553    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw7hx\" (UniqueName: \"kubernetes.io/projected/3278c2e2-7b45-41aa-bd26-941c69ff893b-kube-api-access-nw7hx\") pod \"3278c2e2-7b45-41aa-bd26-941c69ff893b\" (UID: \"3278c2e2-7b45-41aa-bd26-941c69ff893b\") "
Sep 10 19:24:22.339941 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:22.339913    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3278c2e2-7b45-41aa-bd26-941c69ff893b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3278c2e2-7b45-41aa-bd26-941c69ff893b" (UID: "3278c2e2-7b45-41aa-bd26-941c69ff893b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:24:22.341954 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:22.341925    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3278c2e2-7b45-41aa-bd26-941c69ff893b-kube-api-access-nw7hx" (OuterVolumeSpecName: "kube-api-access-nw7hx") pod "3278c2e2-7b45-41aa-bd26-941c69ff893b" (UID: "3278c2e2-7b45-41aa-bd26-941c69ff893b"). InnerVolumeSpecName "kube-api-access-nw7hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:24:22.440744 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:22.440697    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3278c2e2-7b45-41aa-bd26-941c69ff893b-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:22.440744 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:22.440736    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7hx\" (UniqueName: \"kubernetes.io/projected/3278c2e2-7b45-41aa-bd26-941c69ff893b-kube-api-access-nw7hx\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:23.079370 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:23.079329    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:23.079856 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:23.079533    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:23.137527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:23.137398    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"
Sep 10 19:24:23.137527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:23.137459    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j" event={"ID":"3278c2e2-7b45-41aa-bd26-941c69ff893b","Type":"ContainerDied","Data":"ba65b6f92b4862f9d2d4d09de43ed3bb62c6de880cb5ea26b7e52264386dfa55"}
Sep 10 19:24:23.137757 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:23.137538    2573 scope.go:117] "RemoveContainer" containerID="63d8ae8bf9e0f663b02b60d238d71152123d015f493ffd598e166db16fbb0c6a"
Sep 10 19:24:23.211111 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:23.211081    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"]
Sep 10 19:24:23.216630 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:23.216590    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3c7a5e-76548d9cfc-4g99j"]
Sep 10 19:24:24.834069 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:24.834035    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3278c2e2-7b45-41aa-bd26-941c69ff893b" path="/var/lib/kubelet/pods/3278c2e2-7b45-41aa-bd26-941c69ff893b/volumes"
Sep 10 19:24:27.688409 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:27.688323    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"]
Sep 10 19:24:40.453874 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.453840    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"]
Sep 10 19:24:40.454437 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.454089    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn" podUID="ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" containerName="maas-api" containerID="cri-o://1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531" gracePeriod=30
Sep 10 19:24:40.569666 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.569640    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"]
Sep 10 19:24:40.569913 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.569874    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq" podUID="b9e878ec-79f1-44c9-84ea-def49999ec85" containerName="payload-processing" containerID="cri-o://070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea" gracePeriod=30
Sep 10 19:24:40.580513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.580473    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"]
Sep 10 19:24:40.580825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.580783    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg" podUID="3bd01808-3601-4ebf-a121-45dd0f79d8e0" containerName="payload-pre-processing" containerID="cri-o://bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa" gracePeriod=30
Sep 10 19:24:40.715309 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.715235    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:40.820704 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.820669    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-metrics-certs\") pod \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") "
Sep 10 19:24:40.820864 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.820799    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fgxg\" (UniqueName: \"kubernetes.io/projected/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-kube-api-access-2fgxg\") pod \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") "
Sep 10 19:24:40.820864 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.820840    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-openshift-service-ca\") pod \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") "
Sep 10 19:24:40.820982 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.820911    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-maas-api-tls\") pod \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\" (UID: \"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475\") "
Sep 10 19:24:40.821355 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.821319    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" (UID: "ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:24:40.823540 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.823505    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" (UID: "ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:24:40.823645 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.823598    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-kube-api-access-2fgxg" (OuterVolumeSpecName: "kube-api-access-2fgxg") pod "ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" (UID: "ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475"). InnerVolumeSpecName "kube-api-access-2fgxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:24:40.825096 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.825071    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" (UID: "ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:24:40.842826 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.842800    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:40.849893 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.849862    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:40.921864 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.921822    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8ts2\" (UniqueName: \"kubernetes.io/projected/3bd01808-3601-4ebf-a121-45dd0f79d8e0-kube-api-access-v8ts2\") pod \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\" (UID: \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\") "
Sep 10 19:24:40.921864 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.921864    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpjlb\" (UniqueName: \"kubernetes.io/projected/b9e878ec-79f1-44c9-84ea-def49999ec85-kube-api-access-hpjlb\") pod \"b9e878ec-79f1-44c9-84ea-def49999ec85\" (UID: \"b9e878ec-79f1-44c9-84ea-def49999ec85\") "
Sep 10 19:24:40.922125 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.921901    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd01808-3601-4ebf-a121-45dd0f79d8e0-plugins-config-volume\") pod \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\" (UID: \"3bd01808-3601-4ebf-a121-45dd0f79d8e0\") "
Sep 10 19:24:40.922125 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.921962    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e878ec-79f1-44c9-84ea-def49999ec85-plugins-config-volume\") pod \"b9e878ec-79f1-44c9-84ea-def49999ec85\" (UID: \"b9e878ec-79f1-44c9-84ea-def49999ec85\") "
Sep 10 19:24:40.922272 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.922254    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:40.922333 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.922279    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fgxg\" (UniqueName: \"kubernetes.io/projected/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-kube-api-access-2fgxg\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:40.922333 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.922296    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:40.922333 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.922316    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:40.922505 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.922344    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd01808-3601-4ebf-a121-45dd0f79d8e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3bd01808-3601-4ebf-a121-45dd0f79d8e0" (UID: "3bd01808-3601-4ebf-a121-45dd0f79d8e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:24:40.922505 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.922389    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e878ec-79f1-44c9-84ea-def49999ec85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9e878ec-79f1-44c9-84ea-def49999ec85" (UID: "b9e878ec-79f1-44c9-84ea-def49999ec85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:24:40.924034 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.924009    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd01808-3601-4ebf-a121-45dd0f79d8e0-kube-api-access-v8ts2" (OuterVolumeSpecName: "kube-api-access-v8ts2") pod "3bd01808-3601-4ebf-a121-45dd0f79d8e0" (UID: "3bd01808-3601-4ebf-a121-45dd0f79d8e0"). InnerVolumeSpecName "kube-api-access-v8ts2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:24:40.924187 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:40.924100    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e878ec-79f1-44c9-84ea-def49999ec85-kube-api-access-hpjlb" (OuterVolumeSpecName: "kube-api-access-hpjlb") pod "b9e878ec-79f1-44c9-84ea-def49999ec85" (UID: "b9e878ec-79f1-44c9-84ea-def49999ec85"). InnerVolumeSpecName "kube-api-access-hpjlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:24:41.023310 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.023283    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ts2\" (UniqueName: \"kubernetes.io/projected/3bd01808-3601-4ebf-a121-45dd0f79d8e0-kube-api-access-v8ts2\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:41.023310 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.023309    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpjlb\" (UniqueName: \"kubernetes.io/projected/b9e878ec-79f1-44c9-84ea-def49999ec85-kube-api-access-hpjlb\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:41.023497 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.023319    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bd01808-3601-4ebf-a121-45dd0f79d8e0-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:41.023497 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.023330    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9e878ec-79f1-44c9-84ea-def49999ec85-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:41.236952 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.236921    2573 generic.go:356] "Generic (PLEG): container finished" podID="b9e878ec-79f1-44c9-84ea-def49999ec85" containerID="070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea" exitCode=0
Sep 10 19:24:41.237122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.236970    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"
Sep 10 19:24:41.237122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.237001    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq" event={"ID":"b9e878ec-79f1-44c9-84ea-def49999ec85","Type":"ContainerDied","Data":"070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea"}
Sep 10 19:24:41.237122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.237060    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq" event={"ID":"b9e878ec-79f1-44c9-84ea-def49999ec85","Type":"ContainerDied","Data":"149d2643c91645dabe3e9b9f00beaa99b52dd4c012ecb7ce5f705997f4e356f3"}
Sep 10 19:24:41.237122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.237091    2573 scope.go:117] "RemoveContainer" containerID="070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea"
Sep 10 19:24:41.238435 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.238420    2573 generic.go:356] "Generic (PLEG): container finished" podID="ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" containerID="1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531" exitCode=0
Sep 10 19:24:41.238578 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.238452    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"
Sep 10 19:24:41.238578 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.238490    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn" event={"ID":"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475","Type":"ContainerDied","Data":"1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531"}
Sep 10 19:24:41.238578 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.238538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn" event={"ID":"ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475","Type":"ContainerDied","Data":"2c161f3331f9c776187ff1e1bb47d84fc966db052d75aeaacc86b2365ca030ff"}
Sep 10 19:24:41.240203 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.240179    2573 generic.go:356] "Generic (PLEG): container finished" podID="3bd01808-3601-4ebf-a121-45dd0f79d8e0" containerID="bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa" exitCode=0
Sep 10 19:24:41.240321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.240213    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"
Sep 10 19:24:41.240321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.240235    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg" event={"ID":"3bd01808-3601-4ebf-a121-45dd0f79d8e0","Type":"ContainerDied","Data":"bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa"}
Sep 10 19:24:41.240321 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.240269    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg" event={"ID":"3bd01808-3601-4ebf-a121-45dd0f79d8e0","Type":"ContainerDied","Data":"71943b4ad6acc2ff750b4f4f89e5e1f3b8f58d4e592f3a2c9db72fb147d3aa41"}
Sep 10 19:24:41.247617 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.247599    2573 scope.go:117] "RemoveContainer" containerID="070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea"
Sep 10 19:24:41.247939 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:24:41.247922    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea\": container with ID starting with 070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea not found: ID does not exist" containerID="070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea"
Sep 10 19:24:41.248008 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.247948    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea"} err="failed to get container status \"070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea\": rpc error: code = NotFound desc = could not find container \"070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea\": container with ID starting with 070c708ddb79c54b285d28afe56e2114539f0c52b38aeb8c7ef54b4e7b6eb6ea not found: ID does not exist"
Sep 10 19:24:41.248008 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.247964    2573 scope.go:117] "RemoveContainer" containerID="1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531"
Sep 10 19:24:41.256130 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.256113    2573 scope.go:117] "RemoveContainer" containerID="1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531"
Sep 10 19:24:41.256541 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:24:41.256476    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531\": container with ID starting with 1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531 not found: ID does not exist" containerID="1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531"
Sep 10 19:24:41.256541 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.256509    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531"} err="failed to get container status \"1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531\": rpc error: code = NotFound desc = could not find container \"1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531\": container with ID starting with 1b520828da4bbc4ecb4efd1272957e0262e5631f190de38d6a6bf16257eb1531 not found: ID does not exist"
Sep 10 19:24:41.256541 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.256531    2573 scope.go:117] "RemoveContainer" containerID="bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa"
Sep 10 19:24:41.259802 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.259781    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"]
Sep 10 19:24:41.261630 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.261610    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a888f7aa-579f4f4cc-r4dhn"]
Sep 10 19:24:41.265358 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.265342    2573 scope.go:117] "RemoveContainer" containerID="bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa"
Sep 10 19:24:41.265661 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:24:41.265644    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa\": container with ID starting with bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa not found: ID does not exist" containerID="bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa"
Sep 10 19:24:41.265703 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.265667    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa"} err="failed to get container status \"bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa\": rpc error: code = NotFound desc = could not find container \"bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa\": container with ID starting with bec0d7200711072d581bfc5121e8b60af795af070864fc06b6c27939796ea8fa not found: ID does not exist"
Sep 10 19:24:41.282553 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.282489    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"]
Sep 10 19:24:41.291294 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.291268    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a888f7aa-64c6bb58b5-mjdfq"]
Sep 10 19:24:41.306725 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.306701    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"]
Sep 10 19:24:41.318727 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:41.318704    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a888f7aa-6685ddb778-tmlxg"]
Sep 10 19:24:42.709210 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:42.709142    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf" podUID="2da117fc-89e5-42a6-aaab-a3294943e5e3" containerName="main" containerID="cri-o://83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2" gracePeriod=45
Sep 10 19:24:42.833673 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:42.833644    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bd01808-3601-4ebf-a121-45dd0f79d8e0" path="/var/lib/kubelet/pods/3bd01808-3601-4ebf-a121-45dd0f79d8e0/volumes"
Sep 10 19:24:42.834032 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:42.834016    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e878ec-79f1-44c9-84ea-def49999ec85" path="/var/lib/kubelet/pods/b9e878ec-79f1-44c9-84ea-def49999ec85/volumes"
Sep 10 19:24:42.834454 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:42.834440    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" path="/var/lib/kubelet/pods/ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475/volumes"
Sep 10 19:24:42.962971 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:42.962918    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:24:43.043420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.043393    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-model-cache\") pod \"2da117fc-89e5-42a6-aaab-a3294943e5e3\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") "
Sep 10 19:24:43.043603 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.043445    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da117fc-89e5-42a6-aaab-a3294943e5e3-tls-certs\") pod \"2da117fc-89e5-42a6-aaab-a3294943e5e3\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") "
Sep 10 19:24:43.043603 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.043483    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-tmp-dir\") pod \"2da117fc-89e5-42a6-aaab-a3294943e5e3\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") "
Sep 10 19:24:43.043603 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.043505    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-dshm\") pod \"2da117fc-89e5-42a6-aaab-a3294943e5e3\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") "
Sep 10 19:24:43.043603 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.043549    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnnm4\" (UniqueName: \"kubernetes.io/projected/2da117fc-89e5-42a6-aaab-a3294943e5e3-kube-api-access-qnnm4\") pod \"2da117fc-89e5-42a6-aaab-a3294943e5e3\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") "
Sep 10 19:24:43.043603 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.043583    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-home\") pod \"2da117fc-89e5-42a6-aaab-a3294943e5e3\" (UID: \"2da117fc-89e5-42a6-aaab-a3294943e5e3\") "
Sep 10 19:24:43.043911 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.043679    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "2da117fc-89e5-42a6-aaab-a3294943e5e3" (UID: "2da117fc-89e5-42a6-aaab-a3294943e5e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:24:43.043911 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.043805    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2da117fc-89e5-42a6-aaab-a3294943e5e3" (UID: "2da117fc-89e5-42a6-aaab-a3294943e5e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:24:43.044024 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.043985    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-home" (OuterVolumeSpecName: "home") pod "2da117fc-89e5-42a6-aaab-a3294943e5e3" (UID: "2da117fc-89e5-42a6-aaab-a3294943e5e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:24:43.044083 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.044049    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:43.044083 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.044062    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:43.046615 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.046587    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-dshm" (OuterVolumeSpecName: "dshm") pod "2da117fc-89e5-42a6-aaab-a3294943e5e3" (UID: "2da117fc-89e5-42a6-aaab-a3294943e5e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:24:43.046730 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.046587    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da117fc-89e5-42a6-aaab-a3294943e5e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2da117fc-89e5-42a6-aaab-a3294943e5e3" (UID: "2da117fc-89e5-42a6-aaab-a3294943e5e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:24:43.046730 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.046663    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da117fc-89e5-42a6-aaab-a3294943e5e3-kube-api-access-qnnm4" (OuterVolumeSpecName: "kube-api-access-qnnm4") pod "2da117fc-89e5-42a6-aaab-a3294943e5e3" (UID: "2da117fc-89e5-42a6-aaab-a3294943e5e3"). InnerVolumeSpecName "kube-api-access-qnnm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:24:43.145341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.145290    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:43.145341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.145333    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnnm4\" (UniqueName: \"kubernetes.io/projected/2da117fc-89e5-42a6-aaab-a3294943e5e3-kube-api-access-qnnm4\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:43.145341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.145344    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2da117fc-89e5-42a6-aaab-a3294943e5e3-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:43.145341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.145352    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da117fc-89e5-42a6-aaab-a3294943e5e3-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:24:43.255279 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.255186    2573 generic.go:356] "Generic (PLEG): container finished" podID="2da117fc-89e5-42a6-aaab-a3294943e5e3" containerID="83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2" exitCode=0
Sep 10 19:24:43.255279 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.255230    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"
Sep 10 19:24:43.255505 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.255281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf" event={"ID":"2da117fc-89e5-42a6-aaab-a3294943e5e3","Type":"ContainerDied","Data":"83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2"}
Sep 10 19:24:43.255505 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.255336    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf" event={"ID":"2da117fc-89e5-42a6-aaab-a3294943e5e3","Type":"ContainerDied","Data":"418518a972563174e295a473b6e92ecadcd80179bf84bc69206cad2b7aed5c71"}
Sep 10 19:24:43.255505 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.255360    2573 scope.go:117] "RemoveContainer" containerID="83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2"
Sep 10 19:24:43.268384 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.265817    2573 scope.go:117] "RemoveContainer" containerID="83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2"
Sep 10 19:24:43.269096 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:24:43.269059    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2\": container with ID starting with 83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2 not found: ID does not exist" containerID="83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2"
Sep 10 19:24:43.269249 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.269096    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2"} err="failed to get container status \"83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2\": rpc error: code = NotFound desc = could not find container \"83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2\": container with ID starting with 83aaa7e4bf5ed2cf1de0f0393ce585eb70c800fddda5006ca8fe4749964565a2 not found: ID does not exist"
Sep 10 19:24:43.284632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.284601    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"]
Sep 10 19:24:43.286226 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:43.286206    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-3c7a5e/autores-model-3c7a5e-kserve-59fc86fc6c-7fhrf"]
Sep 10 19:24:44.833663 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:24:44.833627    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da117fc-89e5-42a6-aaab-a3294943e5e3" path="/var/lib/kubelet/pods/2da117fc-89e5-42a6-aaab-a3294943e5e3/volumes"
Sep 10 19:25:02.886425 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886394    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"]
Sep 10 19:25:02.886792 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886780    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" containerName="maas-api"
Sep 10 19:25:02.886841 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886793    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" containerName="maas-api"
Sep 10 19:25:02.886841 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886810    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da117fc-89e5-42a6-aaab-a3294943e5e3" containerName="main"
Sep 10 19:25:02.886841 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886816    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da117fc-89e5-42a6-aaab-a3294943e5e3" containerName="main"
Sep 10 19:25:02.886841 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886831    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e878ec-79f1-44c9-84ea-def49999ec85" containerName="payload-processing"
Sep 10 19:25:02.886841 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886837    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e878ec-79f1-44c9-84ea-def49999ec85" containerName="payload-processing"
Sep 10 19:25:02.886987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886847    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3278c2e2-7b45-41aa-bd26-941c69ff893b" containerName="payload-processing"
Sep 10 19:25:02.886987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886852    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3278c2e2-7b45-41aa-bd26-941c69ff893b" containerName="payload-processing"
Sep 10 19:25:02.886987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886859    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bd01808-3601-4ebf-a121-45dd0f79d8e0" containerName="payload-pre-processing"
Sep 10 19:25:02.886987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886865    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bd01808-3601-4ebf-a121-45dd0f79d8e0" containerName="payload-pre-processing"
Sep 10 19:25:02.886987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886916    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3278c2e2-7b45-41aa-bd26-941c69ff893b" containerName="payload-processing"
Sep 10 19:25:02.886987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886924    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da117fc-89e5-42a6-aaab-a3294943e5e3" containerName="main"
Sep 10 19:25:02.886987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886930    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bd01808-3601-4ebf-a121-45dd0f79d8e0" containerName="payload-pre-processing"
Sep 10 19:25:02.886987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886938    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e878ec-79f1-44c9-84ea-def49999ec85" containerName="payload-processing"
Sep 10 19:25:02.886987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.886945    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef02976e-1c47-4ab4-b4fb-f5b6e6cd6475" containerName="maas-api"
Sep 10 19:25:02.889488 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.889472    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:02.893582 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.893561    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-fcaf1295-kserve-self-signed-certs\""
Sep 10 19:25:02.933001 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.932968    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-tmp-dir\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:02.933141 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.933004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7fa6e3-abba-4f13-8d13-1128294b73df-tls-certs\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:02.933141 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.933096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltw5h\" (UniqueName: \"kubernetes.io/projected/ab7fa6e3-abba-4f13-8d13-1128294b73df-kube-api-access-ltw5h\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:02.933141 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.933123    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-home\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:02.933289 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.933144    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-dshm\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:02.933289 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:02.933203    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-model-cache\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.020431 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.020395    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"]
Sep 10 19:25:03.034287 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.034259    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-tmp-dir\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.034287 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.034290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7fa6e3-abba-4f13-8d13-1128294b73df-tls-certs\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.034519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.034322    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltw5h\" (UniqueName: \"kubernetes.io/projected/ab7fa6e3-abba-4f13-8d13-1128294b73df-kube-api-access-ltw5h\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.034519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.034337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-home\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.034519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.034353    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-dshm\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.034519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.034479    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-model-cache\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.034745 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.034715    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-tmp-dir\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.034806 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.034789    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-home\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.034892 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.034859    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-model-cache\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.036746 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.036724    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-dshm\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.036987 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.036969    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7fa6e3-abba-4f13-8d13-1128294b73df-tls-certs\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.047019 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.046993    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltw5h\" (UniqueName: \"kubernetes.io/projected/ab7fa6e3-abba-4f13-8d13-1128294b73df-kube-api-access-ltw5h\") pod \"e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") " pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.198969 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.198872    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:03.484841 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:03.484815    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"]
Sep 10 19:25:03.486366 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:25:03.486338    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab7fa6e3_abba_4f13_8d13_1128294b73df.slice/crio-c298ff5dd7276e94b52cc4b4420e680ac855efcc4193389ec234315d1517fb0f WatchSource:0}: Error finding container c298ff5dd7276e94b52cc4b4420e680ac855efcc4193389ec234315d1517fb0f: Status 404 returned error can't find the container with id c298ff5dd7276e94b52cc4b4420e680ac855efcc4193389ec234315d1517fb0f
Sep 10 19:25:04.366925 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:04.366868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd" event={"ID":"ab7fa6e3-abba-4f13-8d13-1128294b73df","Type":"ContainerStarted","Data":"3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a"}
Sep 10 19:25:04.366925 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:04.366930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd" event={"ID":"ab7fa6e3-abba-4f13-8d13-1128294b73df","Type":"ContainerStarted","Data":"c298ff5dd7276e94b52cc4b4420e680ac855efcc4193389ec234315d1517fb0f"}
Sep 10 19:25:04.390863 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:04.390801    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd" podStartSLOduration=2.390786095 podStartE2EDuration="2.390786095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:25:04.389698439 +0000 UTC m=+840.049495663" watchObservedRunningTime="2026-09-10 19:25:04.390786095 +0000 UTC m=+840.050583314"
Sep 10 19:25:13.200038 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:13.200000    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:13.200518 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:13.200054    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:13.213202 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:13.213172    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:13.423552 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:13.423526    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:16.600660 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.600618    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"]
Sep 10 19:25:16.606962 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.606933    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.609826 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.609798    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-717161-serving-cert\""
Sep 10 19:25:16.610505 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.610477    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lw7bv\""
Sep 10 19:25:16.610619 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.610474    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 19:25:16.617713 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.617679    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"]
Sep 10 19:25:16.656329 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.656290    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-metrics-certs\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.656512 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.656357    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57bz\" (UniqueName: \"kubernetes.io/projected/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-kube-api-access-w57bz\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.656512 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.656386    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-openshift-service-ca\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.656512 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.656475    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-maas-api-tls\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.757668 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.757619    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57bz\" (UniqueName: \"kubernetes.io/projected/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-kube-api-access-w57bz\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.757885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.757679    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-openshift-service-ca\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.757885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.757763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-maas-api-tls\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.757885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.757808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-metrics-certs\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.758087 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:25:16.757916    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-717161-serving-cert: secret "maas-api-e2e-explicit-717161-serving-cert" not found
Sep 10 19:25:16.758087 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:25:16.758001    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-maas-api-tls podName:833a0591-5dda-4c93-a585-9b7f0bfe2b2e nodeName:}" failed. No retries permitted until 2026-09-10 19:25:17.257977515 +0000 UTC m=+852.917774703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-maas-api-tls") pod "maas-api-e2e-explicit-717161-7479d995f7-dpzp5" (UID: "833a0591-5dda-4c93-a585-9b7f0bfe2b2e") : secret "maas-api-e2e-explicit-717161-serving-cert" not found
Sep 10 19:25:16.758531 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.758512    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-openshift-service-ca\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.760434 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.760412    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-metrics-certs\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:16.768090 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:16.768056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57bz\" (UniqueName: \"kubernetes.io/projected/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-kube-api-access-w57bz\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:17.263726 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.263687    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-maas-api-tls\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:17.266341 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.266309    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-maas-api-tls\") pod \"maas-api-e2e-explicit-717161-7479d995f7-dpzp5\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:17.304289 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.304251    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"]
Sep 10 19:25:17.308284 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.308252    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:17.310850 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.310826    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-717161-dockercfg-6c2dl\""
Sep 10 19:25:17.310974 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.310826    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-717161\""
Sep 10 19:25:17.319468 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.319442    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"]
Sep 10 19:25:17.364939 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.364901    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nrf\" (UniqueName: \"kubernetes.io/projected/a18af411-b51d-43ce-a28f-350acc21fb11-kube-api-access-49nrf\") pod \"payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd\" (UID: \"a18af411-b51d-43ce-a28f-350acc21fb11\") " pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:17.365182 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.365026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18af411-b51d-43ce-a28f-350acc21fb11-plugins-config-volume\") pod \"payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd\" (UID: \"a18af411-b51d-43ce-a28f-350acc21fb11\") " pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:17.373068 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.373036    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"]
Sep 10 19:25:17.376601 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.376585    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:17.385618 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.385591    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"]
Sep 10 19:25:17.466136 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.466100    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbnnn\" (UniqueName: \"kubernetes.io/projected/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-kube-api-access-wbnnn\") pod \"payload-pre-processing-e2e-explicit-717161-76879558db-blsnv\" (UID: \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:17.466136 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.466142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nrf\" (UniqueName: \"kubernetes.io/projected/a18af411-b51d-43ce-a28f-350acc21fb11-kube-api-access-49nrf\") pod \"payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd\" (UID: \"a18af411-b51d-43ce-a28f-350acc21fb11\") " pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:17.466444 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.466264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18af411-b51d-43ce-a28f-350acc21fb11-plugins-config-volume\") pod \"payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd\" (UID: \"a18af411-b51d-43ce-a28f-350acc21fb11\") " pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:17.466444 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.466327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-717161-76879558db-blsnv\" (UID: \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:17.466970 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.466947    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18af411-b51d-43ce-a28f-350acc21fb11-plugins-config-volume\") pod \"payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd\" (UID: \"a18af411-b51d-43ce-a28f-350acc21fb11\") " pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:17.488031 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.488002    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nrf\" (UniqueName: \"kubernetes.io/projected/a18af411-b51d-43ce-a28f-350acc21fb11-kube-api-access-49nrf\") pod \"payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd\" (UID: \"a18af411-b51d-43ce-a28f-350acc21fb11\") " pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:17.521181 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.521130    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:17.567636 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.567595    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-717161-76879558db-blsnv\" (UID: \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:17.567851 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.567690    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbnnn\" (UniqueName: \"kubernetes.io/projected/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-kube-api-access-wbnnn\") pod \"payload-pre-processing-e2e-explicit-717161-76879558db-blsnv\" (UID: \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:17.568760 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.568735    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-717161-76879558db-blsnv\" (UID: \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:17.578063 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.578035    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbnnn\" (UniqueName: \"kubernetes.io/projected/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-kube-api-access-wbnnn\") pod \"payload-pre-processing-e2e-explicit-717161-76879558db-blsnv\" (UID: \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:17.620294 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.620249    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:17.689795 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.689717    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:17.800902 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.800876    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"]
Sep 10 19:25:17.803559 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:25:17.803530    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833a0591_5dda_4c93_a585_9b7f0bfe2b2e.slice/crio-5d11a7f32ec36d41e5b3524ecec27a2544f630b25a9d6451d228759020dc79f8 WatchSource:0}: Error finding container 5d11a7f32ec36d41e5b3524ecec27a2544f630b25a9d6451d228759020dc79f8: Status 404 returned error can't find the container with id 5d11a7f32ec36d41e5b3524ecec27a2544f630b25a9d6451d228759020dc79f8
Sep 10 19:25:17.905785 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.905717    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"]
Sep 10 19:25:17.908202 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:25:17.908171    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18af411_b51d_43ce_a28f_350acc21fb11.slice/crio-35f25f9da2aa449b160a2aa9f4b5c0039f44523b93395ef3934fca749d4ebd40 WatchSource:0}: Error finding container 35f25f9da2aa449b160a2aa9f4b5c0039f44523b93395ef3934fca749d4ebd40: Status 404 returned error can't find the container with id 35f25f9da2aa449b160a2aa9f4b5c0039f44523b93395ef3934fca749d4ebd40
Sep 10 19:25:17.970786 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:17.970761    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"]
Sep 10 19:25:17.973093 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:25:17.973064    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3fe555_2097_4a7d_9ba6_5cf5451f39fd.slice/crio-f9c27a2a591fede66539486e12a1b322d149422f5e5cfd61a3c99297f6a912b0 WatchSource:0}: Error finding container f9c27a2a591fede66539486e12a1b322d149422f5e5cfd61a3c99297f6a912b0: Status 404 returned error can't find the container with id f9c27a2a591fede66539486e12a1b322d149422f5e5cfd61a3c99297f6a912b0
Sep 10 19:25:18.439081 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.438976    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd" event={"ID":"a18af411-b51d-43ce-a28f-350acc21fb11","Type":"ContainerStarted","Data":"0e470cbe56b1d9a0a0c45eea3099f05859ef5a33c405a3f85bfcb78303b4c52a"}
Sep 10 19:25:18.439081 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.439042    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd" event={"ID":"a18af411-b51d-43ce-a28f-350acc21fb11","Type":"ContainerStarted","Data":"35f25f9da2aa449b160a2aa9f4b5c0039f44523b93395ef3934fca749d4ebd40"}
Sep 10 19:25:18.439356 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.439082    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:18.441057 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.441023    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5" event={"ID":"833a0591-5dda-4c93-a585-9b7f0bfe2b2e","Type":"ContainerStarted","Data":"b47e711be5d6a92055f2366363167a918531e6e987eaa97271b49b7508b030dd"}
Sep 10 19:25:18.441215 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.441083    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5" event={"ID":"833a0591-5dda-4c93-a585-9b7f0bfe2b2e","Type":"ContainerStarted","Data":"5d11a7f32ec36d41e5b3524ecec27a2544f630b25a9d6451d228759020dc79f8"}
Sep 10 19:25:18.441215 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.441118    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:18.442738 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.442713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv" event={"ID":"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd","Type":"ContainerStarted","Data":"a1cdd2f8d86d66bde5f5ed5f48549e068ce5cbfab2dcb691f6d6ed9de8d0d94d"}
Sep 10 19:25:18.442851 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.442755    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:18.442851 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.442773    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv" event={"ID":"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd","Type":"ContainerStarted","Data":"f9c27a2a591fede66539486e12a1b322d149422f5e5cfd61a3c99297f6a912b0"}
Sep 10 19:25:18.462301 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.462245    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd" podStartSLOduration=1.46222762 podStartE2EDuration="1.46222762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:25:18.460702047 +0000 UTC m=+854.120499243" watchObservedRunningTime="2026-09-10 19:25:18.46222762 +0000 UTC m=+854.122024818"
Sep 10 19:25:18.482033 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.481971    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5" podStartSLOduration=2.067875257 podStartE2EDuration="2.481952195s" totalImagesPullingTime="414.076938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:25:18.480389421 +0000 UTC m=+854.140186615" watchObservedRunningTime="2026-09-10 19:25:18.481952195 +0000 UTC m=+854.141749393"
Sep 10 19:25:18.498557 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:18.498483    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv" podStartSLOduration=1.498458044 podStartE2EDuration="1.498458044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:25:18.497377344 +0000 UTC m=+854.157174537" watchObservedRunningTime="2026-09-10 19:25:18.498458044 +0000 UTC m=+854.158255238"
Sep 10 19:25:23.021747 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:23.021715    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"]
Sep 10 19:25:24.453771 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:24.453730    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:25:29.450288 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:29.450255    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:25:29.450700 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:29.450491    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:25:38.042236 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.042195    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd" podUID="ab7fa6e3-abba-4f13-8d13-1128294b73df" containerName="main" containerID="cri-o://3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a" gracePeriod=45
Sep 10 19:25:38.304227 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.304140    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:38.482562 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.482518    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltw5h\" (UniqueName: \"kubernetes.io/projected/ab7fa6e3-abba-4f13-8d13-1128294b73df-kube-api-access-ltw5h\") pod \"ab7fa6e3-abba-4f13-8d13-1128294b73df\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") "
Sep 10 19:25:38.482562 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.482565    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-dshm\") pod \"ab7fa6e3-abba-4f13-8d13-1128294b73df\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") "
Sep 10 19:25:38.482777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.482610    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-home\") pod \"ab7fa6e3-abba-4f13-8d13-1128294b73df\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") "
Sep 10 19:25:38.482777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.482717    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7fa6e3-abba-4f13-8d13-1128294b73df-tls-certs\") pod \"ab7fa6e3-abba-4f13-8d13-1128294b73df\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") "
Sep 10 19:25:38.482777 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.482756    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-tmp-dir\") pod \"ab7fa6e3-abba-4f13-8d13-1128294b73df\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") "
Sep 10 19:25:38.482920 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.482787    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-model-cache\") pod \"ab7fa6e3-abba-4f13-8d13-1128294b73df\" (UID: \"ab7fa6e3-abba-4f13-8d13-1128294b73df\") "
Sep 10 19:25:38.483000 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.482960    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-home" (OuterVolumeSpecName: "home") pod "ab7fa6e3-abba-4f13-8d13-1128294b73df" (UID: "ab7fa6e3-abba-4f13-8d13-1128294b73df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:38.483127 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.483079    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab7fa6e3-abba-4f13-8d13-1128294b73df" (UID: "ab7fa6e3-abba-4f13-8d13-1128294b73df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:38.483205 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.483173    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab7fa6e3-abba-4f13-8d13-1128294b73df" (UID: "ab7fa6e3-abba-4f13-8d13-1128294b73df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:38.483205 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.483188    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:25:38.484967 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.484944    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-dshm" (OuterVolumeSpecName: "dshm") pod "ab7fa6e3-abba-4f13-8d13-1128294b73df" (UID: "ab7fa6e3-abba-4f13-8d13-1128294b73df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:38.485374 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.485342    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7fa6e3-abba-4f13-8d13-1128294b73df-kube-api-access-ltw5h" (OuterVolumeSpecName: "kube-api-access-ltw5h") pod "ab7fa6e3-abba-4f13-8d13-1128294b73df" (UID: "ab7fa6e3-abba-4f13-8d13-1128294b73df"). InnerVolumeSpecName "kube-api-access-ltw5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:25:38.485374 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.485366    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7fa6e3-abba-4f13-8d13-1128294b73df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab7fa6e3-abba-4f13-8d13-1128294b73df" (UID: "ab7fa6e3-abba-4f13-8d13-1128294b73df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:25:38.549099 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.549056    2573 generic.go:356] "Generic (PLEG): container finished" podID="ab7fa6e3-abba-4f13-8d13-1128294b73df" containerID="3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a" exitCode=0
Sep 10 19:25:38.549302 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.549122    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"
Sep 10 19:25:38.549302 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.549139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd" event={"ID":"ab7fa6e3-abba-4f13-8d13-1128294b73df","Type":"ContainerDied","Data":"3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a"}
Sep 10 19:25:38.549302 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.549198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd" event={"ID":"ab7fa6e3-abba-4f13-8d13-1128294b73df","Type":"ContainerDied","Data":"c298ff5dd7276e94b52cc4b4420e680ac855efcc4193389ec234315d1517fb0f"}
Sep 10 19:25:38.549302 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.549215    2573 scope.go:117] "RemoveContainer" containerID="3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a"
Sep 10 19:25:38.561518 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.561497    2573 scope.go:117] "RemoveContainer" containerID="3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a"
Sep 10 19:25:38.561907 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:25:38.561889    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a\": container with ID starting with 3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a not found: ID does not exist" containerID="3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a"
Sep 10 19:25:38.561956 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.561917    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a"} err="failed to get container status \"3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a\": rpc error: code = NotFound desc = could not find container \"3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a\": container with ID starting with 3a0c799343252554b24c1fe5a080cdb72db1597b1ac67603c588defa5b40364a not found: ID does not exist"
Sep 10 19:25:38.571531 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.571498    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"]
Sep 10 19:25:38.576961 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.576934    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-fcaf1295-kserve-7cb688fb8-n94xd"]
Sep 10 19:25:38.584270 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.584244    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab7fa6e3-abba-4f13-8d13-1128294b73df-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:25:38.584270 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.584268    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:25:38.584270 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.584276    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:25:38.584469 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.584284    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltw5h\" (UniqueName: \"kubernetes.io/projected/ab7fa6e3-abba-4f13-8d13-1128294b73df-kube-api-access-ltw5h\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:25:38.584469 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.584293    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab7fa6e3-abba-4f13-8d13-1128294b73df-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:25:38.833541 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:38.833465    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7fa6e3-abba-4f13-8d13-1128294b73df" path="/var/lib/kubelet/pods/ab7fa6e3-abba-4f13-8d13-1128294b73df/volumes"
Sep 10 19:25:51.711332 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.711298    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"]
Sep 10 19:25:51.711844 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.711762    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7fa6e3-abba-4f13-8d13-1128294b73df" containerName="main"
Sep 10 19:25:51.711844 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.711784    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7fa6e3-abba-4f13-8d13-1128294b73df" containerName="main"
Sep 10 19:25:51.711946 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.711865    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7fa6e3-abba-4f13-8d13-1128294b73df" containerName="main"
Sep 10 19:25:51.716565 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.716542    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.720296 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.720260    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b960a2\"/\"openshift-service-ca.crt\""
Sep 10 19:25:51.720296 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.720260    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b960a2\"/\"default-dockercfg-kk2qz\""
Sep 10 19:25:51.720296 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.720294    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b960a2\"/\"kube-root-ca.crt\""
Sep 10 19:25:51.720564 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.720261    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b960a2\"/\"test-model-b960a2-kserve-self-signed-certs\""
Sep 10 19:25:51.727033 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.726999    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"]
Sep 10 19:25:51.799577 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.799543    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-tmp-dir\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.799729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.799594    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-dshm\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.799729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.799631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2zl\" (UniqueName: \"kubernetes.io/projected/33d274f0-bda5-4bdd-8231-6fa19c481910-kube-api-access-pf2zl\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.799729 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.799676    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-home\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.799852 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.799776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d274f0-bda5-4bdd-8231-6fa19c481910-tls-certs\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.799852 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.799806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-model-cache\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.901070 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.901024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-home\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.901284 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.901134    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d274f0-bda5-4bdd-8231-6fa19c481910-tls-certs\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.901284 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.901205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-model-cache\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.901284 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.901230    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-tmp-dir\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.901556 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.901526    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-home\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.901556 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.901537    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-dshm\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.901766 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.901601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2zl\" (UniqueName: \"kubernetes.io/projected/33d274f0-bda5-4bdd-8231-6fa19c481910-kube-api-access-pf2zl\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.901766 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.901624    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-tmp-dir\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.901766 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.901552    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-model-cache\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.903988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.903961    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-dshm\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.904368 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.904344    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d274f0-bda5-4bdd-8231-6fa19c481910-tls-certs\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:51.916519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:51.916486    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2zl\" (UniqueName: \"kubernetes.io/projected/33d274f0-bda5-4bdd-8231-6fa19c481910-kube-api-access-pf2zl\") pod \"test-model-b960a2-kserve-57884d8585-pfrjv\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") " pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:52.051834 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:52.051800    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:25:52.337818 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:52.337723    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"]
Sep 10 19:25:52.340716 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:25:52.340686    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33d274f0_bda5_4bdd_8231_6fa19c481910.slice/crio-fdee38bd2b7d676866107e7f527b1a63750c5b1a02dfe5c39112adf096da83ed WatchSource:0}: Error finding container fdee38bd2b7d676866107e7f527b1a63750c5b1a02dfe5c39112adf096da83ed: Status 404 returned error can't find the container with id fdee38bd2b7d676866107e7f527b1a63750c5b1a02dfe5c39112adf096da83ed
Sep 10 19:25:52.627912 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:52.627806    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv" event={"ID":"33d274f0-bda5-4bdd-8231-6fa19c481910","Type":"ContainerStarted","Data":"479899936b66cc06c3dd30c3adfec8028709ec22178b9d36891f76b1973af05c"}
Sep 10 19:25:52.627912 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:52.627851    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv" event={"ID":"33d274f0-bda5-4bdd-8231-6fa19c481910","Type":"ContainerStarted","Data":"fdee38bd2b7d676866107e7f527b1a63750c5b1a02dfe5c39112adf096da83ed"}
Sep 10 19:25:52.646632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:25:52.646578    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv" podStartSLOduration=1.646558601 podStartE2EDuration="1.646558601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:25:52.645144139 +0000 UTC m=+888.304941332" watchObservedRunningTime="2026-09-10 19:25:52.646558601 +0000 UTC m=+888.306355794"
Sep 10 19:26:02.052464 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:02.052418    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:26:02.052869 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:02.052517    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:26:02.065407 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:02.065376    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:26:02.690357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:02.690331    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:26:22.698377 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.698337    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"]
Sep 10 19:26:22.703366 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.703344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.705757 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.705737    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c4487c09-serving-cert\""
Sep 10 19:26:22.713995 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.713970    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"]
Sep 10 19:26:22.803437 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.803402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf704802-21cf-4479-ac5a-efe4e5423849-openshift-service-ca\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.803609 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.803531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8cr\" (UniqueName: \"kubernetes.io/projected/bf704802-21cf-4479-ac5a-efe4e5423849-kube-api-access-rt8cr\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.803609 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.803581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-maas-api-tls\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.803609 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.803608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-metrics-certs\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.905143 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.905097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8cr\" (UniqueName: \"kubernetes.io/projected/bf704802-21cf-4479-ac5a-efe4e5423849-kube-api-access-rt8cr\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.905387 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.905198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-maas-api-tls\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.905387 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.905237    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-metrics-certs\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.905387 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.905294    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf704802-21cf-4479-ac5a-efe4e5423849-openshift-service-ca\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.905387 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:26:22.905339    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-serving-cert: secret "maas-api-e2e-mt-c4487c09-serving-cert" not found
Sep 10 19:26:22.905591 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:26:22.905418    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-maas-api-tls podName:bf704802-21cf-4479-ac5a-efe4e5423849 nodeName:}" failed. No retries permitted until 2026-09-10 19:26:23.405398471 +0000 UTC m=+919.065195642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-maas-api-tls") pod "maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff" (UID: "bf704802-21cf-4479-ac5a-efe4e5423849") : secret "maas-api-e2e-mt-c4487c09-serving-cert" not found
Sep 10 19:26:22.906039 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.906017    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf704802-21cf-4479-ac5a-efe4e5423849-openshift-service-ca\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.907821 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.907803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-metrics-certs\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:22.917807 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:22.917775    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8cr\" (UniqueName: \"kubernetes.io/projected/bf704802-21cf-4479-ac5a-efe4e5423849-kube-api-access-rt8cr\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:23.410118 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.410068    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-maas-api-tls\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:23.412882 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.412856    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-maas-api-tls\") pod \"maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:23.514344 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.514308    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"]
Sep 10 19:26:23.518493 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.518469    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:23.522607 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.522581    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c4487c09\""
Sep 10 19:26:23.522750 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.522692    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c4487c09-dockercfg-5gq9x\""
Sep 10 19:26:23.532789 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.532759    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"]
Sep 10 19:26:23.615474 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.615435    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:23.712760 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.712713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn\" (UID: \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:23.713274 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.712774    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqx6c\" (UniqueName: \"kubernetes.io/projected/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-kube-api-access-nqx6c\") pod \"payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn\" (UID: \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:23.814416 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.814379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn\" (UID: \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:23.814623 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.814440    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqx6c\" (UniqueName: \"kubernetes.io/projected/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-kube-api-access-nqx6c\") pod \"payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn\" (UID: \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:23.815361 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.815328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn\" (UID: \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:23.823582 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.823559    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqx6c\" (UniqueName: \"kubernetes.io/projected/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-kube-api-access-nqx6c\") pod \"payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn\" (UID: \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:23.831560 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.831535    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:23.893600 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:23.893558    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"]
Sep 10 19:26:24.126438 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:24.126413    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"]
Sep 10 19:26:24.127996 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:26:24.127973    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f787fe_af1e_41a2_ba63_a4f7d6b47fba.slice/crio-6a63729e580b784f5181e025b09c58df33333ccba220694c959b36332b3fc0f5 WatchSource:0}: Error finding container 6a63729e580b784f5181e025b09c58df33333ccba220694c959b36332b3fc0f5: Status 404 returned error can't find the container with id 6a63729e580b784f5181e025b09c58df33333ccba220694c959b36332b3fc0f5
Sep 10 19:26:24.799835 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:24.799790    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn" event={"ID":"59f787fe-af1e-41a2-ba63-a4f7d6b47fba","Type":"ContainerStarted","Data":"bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846"}
Sep 10 19:26:24.800345 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:24.799858    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn" event={"ID":"59f787fe-af1e-41a2-ba63-a4f7d6b47fba","Type":"ContainerStarted","Data":"6a63729e580b784f5181e025b09c58df33333ccba220694c959b36332b3fc0f5"}
Sep 10 19:26:24.800345 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:24.799887    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:24.801709 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:24.801681    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff" event={"ID":"bf704802-21cf-4479-ac5a-efe4e5423849","Type":"ContainerStarted","Data":"b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497"}
Sep 10 19:26:24.801840 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:24.801729    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff" event={"ID":"bf704802-21cf-4479-ac5a-efe4e5423849","Type":"ContainerStarted","Data":"42cc83a3b57e6710c466e242845b60751332b4337fc2954f4a288bc002dfdc14"}
Sep 10 19:26:24.801840 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:24.801762    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:24.820224 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:24.820170    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn" podStartSLOduration=1.820128365 podStartE2EDuration="1.820128365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:24.818198028 +0000 UTC m=+920.477995240" watchObservedRunningTime="2026-09-10 19:26:24.820128365 +0000 UTC m=+920.479925558"
Sep 10 19:26:24.839433 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:24.839371    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff" podStartSLOduration=2.454606301 podStartE2EDuration="2.839352812s" totalImagesPullingTime="384.746511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:24.837514323 +0000 UTC m=+920.497311517" watchObservedRunningTime="2026-09-10 19:26:24.839352812 +0000 UTC m=+920.499150005"
Sep 10 19:26:27.578103 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.578067    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"]
Sep 10 19:26:27.578577 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.578347    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5" podUID="833a0591-5dda-4c93-a585-9b7f0bfe2b2e" containerName="maas-api" containerID="cri-o://b47e711be5d6a92055f2366363167a918531e6e987eaa97271b49b7508b030dd" gracePeriod=30
Sep 10 19:26:27.655489 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.655455    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"]
Sep 10 19:26:27.655746 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.655721    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd" podUID="a18af411-b51d-43ce-a28f-350acc21fb11" containerName="payload-processing" containerID="cri-o://0e470cbe56b1d9a0a0c45eea3099f05859ef5a33c405a3f85bfcb78303b4c52a" gracePeriod=30
Sep 10 19:26:27.666023 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.665992    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"]
Sep 10 19:26:27.666278 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.666253    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv" podUID="5a3fe555-2097-4a7d-9ba6-5cf5451f39fd" containerName="payload-pre-processing" containerID="cri-o://a1cdd2f8d86d66bde5f5ed5f48549e068ce5cbfab2dcb691f6d6ed9de8d0d94d" gracePeriod=30
Sep 10 19:26:27.825344 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.825296    2573 generic.go:356] "Generic (PLEG): container finished" podID="a18af411-b51d-43ce-a28f-350acc21fb11" containerID="0e470cbe56b1d9a0a0c45eea3099f05859ef5a33c405a3f85bfcb78303b4c52a" exitCode=0
Sep 10 19:26:27.825512 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.825339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd" event={"ID":"a18af411-b51d-43ce-a28f-350acc21fb11","Type":"ContainerDied","Data":"0e470cbe56b1d9a0a0c45eea3099f05859ef5a33c405a3f85bfcb78303b4c52a"}
Sep 10 19:26:27.827631 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.827602    2573 generic.go:356] "Generic (PLEG): container finished" podID="833a0591-5dda-4c93-a585-9b7f0bfe2b2e" containerID="b47e711be5d6a92055f2366363167a918531e6e987eaa97271b49b7508b030dd" exitCode=0
Sep 10 19:26:27.827781 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.827660    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5" event={"ID":"833a0591-5dda-4c93-a585-9b7f0bfe2b2e","Type":"ContainerDied","Data":"b47e711be5d6a92055f2366363167a918531e6e987eaa97271b49b7508b030dd"}
Sep 10 19:26:27.829756 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.829680    2573 generic.go:356] "Generic (PLEG): container finished" podID="5a3fe555-2097-4a7d-9ba6-5cf5451f39fd" containerID="a1cdd2f8d86d66bde5f5ed5f48549e068ce5cbfab2dcb691f6d6ed9de8d0d94d" exitCode=0
Sep 10 19:26:27.829865 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.829785    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv" event={"ID":"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd","Type":"ContainerDied","Data":"a1cdd2f8d86d66bde5f5ed5f48549e068ce5cbfab2dcb691f6d6ed9de8d0d94d"}
Sep 10 19:26:27.935305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.935279    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:26:27.949418 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.949385    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:26:27.951065 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.951046    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-metrics-certs\") pod \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") "
Sep 10 19:26:27.951140 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.951118    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w57bz\" (UniqueName: \"kubernetes.io/projected/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-kube-api-access-w57bz\") pod \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") "
Sep 10 19:26:27.951215 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.951140    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-openshift-service-ca\") pod \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") "
Sep 10 19:26:27.951280 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.951229    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-maas-api-tls\") pod \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\" (UID: \"833a0591-5dda-4c93-a585-9b7f0bfe2b2e\") "
Sep 10 19:26:27.951621 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.951576    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "833a0591-5dda-4c93-a585-9b7f0bfe2b2e" (UID: "833a0591-5dda-4c93-a585-9b7f0bfe2b2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:26:27.953517 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.953491    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "833a0591-5dda-4c93-a585-9b7f0bfe2b2e" (UID: "833a0591-5dda-4c93-a585-9b7f0bfe2b2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:26:27.953851 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.953825    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "833a0591-5dda-4c93-a585-9b7f0bfe2b2e" (UID: "833a0591-5dda-4c93-a585-9b7f0bfe2b2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:26:27.953956 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.953925    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-kube-api-access-w57bz" (OuterVolumeSpecName: "kube-api-access-w57bz") pod "833a0591-5dda-4c93-a585-9b7f0bfe2b2e" (UID: "833a0591-5dda-4c93-a585-9b7f0bfe2b2e"). InnerVolumeSpecName "kube-api-access-w57bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:26:27.960865 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:27.960837    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:26:28.051888 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.051854    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49nrf\" (UniqueName: \"kubernetes.io/projected/a18af411-b51d-43ce-a28f-350acc21fb11-kube-api-access-49nrf\") pod \"a18af411-b51d-43ce-a28f-350acc21fb11\" (UID: \"a18af411-b51d-43ce-a28f-350acc21fb11\") "
Sep 10 19:26:28.052053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.051941    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbnnn\" (UniqueName: \"kubernetes.io/projected/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-kube-api-access-wbnnn\") pod \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\" (UID: \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\") "
Sep 10 19:26:28.052053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.051990    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18af411-b51d-43ce-a28f-350acc21fb11-plugins-config-volume\") pod \"a18af411-b51d-43ce-a28f-350acc21fb11\" (UID: \"a18af411-b51d-43ce-a28f-350acc21fb11\") "
Sep 10 19:26:28.052053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.052043    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-plugins-config-volume\") pod \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\" (UID: \"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd\") "
Sep 10 19:26:28.052300 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.052275    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:26:28.052300 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.052296    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w57bz\" (UniqueName: \"kubernetes.io/projected/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-kube-api-access-w57bz\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:26:28.052501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.052310    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:26:28.052501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.052325    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/833a0591-5dda-4c93-a585-9b7f0bfe2b2e-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:26:28.052501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.052452    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18af411-b51d-43ce-a28f-350acc21fb11-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a18af411-b51d-43ce-a28f-350acc21fb11" (UID: "a18af411-b51d-43ce-a28f-350acc21fb11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:26:28.052501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.052483    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a3fe555-2097-4a7d-9ba6-5cf5451f39fd" (UID: "5a3fe555-2097-4a7d-9ba6-5cf5451f39fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:26:28.054088 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.054067    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18af411-b51d-43ce-a28f-350acc21fb11-kube-api-access-49nrf" (OuterVolumeSpecName: "kube-api-access-49nrf") pod "a18af411-b51d-43ce-a28f-350acc21fb11" (UID: "a18af411-b51d-43ce-a28f-350acc21fb11"). InnerVolumeSpecName "kube-api-access-49nrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:26:28.054265 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.054240    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-kube-api-access-wbnnn" (OuterVolumeSpecName: "kube-api-access-wbnnn") pod "5a3fe555-2097-4a7d-9ba6-5cf5451f39fd" (UID: "5a3fe555-2097-4a7d-9ba6-5cf5451f39fd"). InnerVolumeSpecName "kube-api-access-wbnnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:26:28.153027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.152942    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49nrf\" (UniqueName: \"kubernetes.io/projected/a18af411-b51d-43ce-a28f-350acc21fb11-kube-api-access-49nrf\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:26:28.153027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.152969    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbnnn\" (UniqueName: \"kubernetes.io/projected/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-kube-api-access-wbnnn\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:26:28.153027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.152982    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18af411-b51d-43ce-a28f-350acc21fb11-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:26:28.153027 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.152993    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:26:28.846727 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.846631    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"
Sep 10 19:26:28.846727 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.846638    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv" event={"ID":"5a3fe555-2097-4a7d-9ba6-5cf5451f39fd","Type":"ContainerDied","Data":"f9c27a2a591fede66539486e12a1b322d149422f5e5cfd61a3c99297f6a912b0"}
Sep 10 19:26:28.846727 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.846713    2573 scope.go:117] "RemoveContainer" containerID="a1cdd2f8d86d66bde5f5ed5f48549e068ce5cbfab2dcb691f6d6ed9de8d0d94d"
Sep 10 19:26:28.848623 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.848601    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"
Sep 10 19:26:28.848623 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.848613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd" event={"ID":"a18af411-b51d-43ce-a28f-350acc21fb11","Type":"ContainerDied","Data":"35f25f9da2aa449b160a2aa9f4b5c0039f44523b93395ef3934fca749d4ebd40"}
Sep 10 19:26:28.850374 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.850347    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5" event={"ID":"833a0591-5dda-4c93-a585-9b7f0bfe2b2e","Type":"ContainerDied","Data":"5d11a7f32ec36d41e5b3524ecec27a2544f630b25a9d6451d228759020dc79f8"}
Sep 10 19:26:28.850610 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.850390    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"
Sep 10 19:26:28.859224 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.859202    2573 scope.go:117] "RemoveContainer" containerID="0e470cbe56b1d9a0a0c45eea3099f05859ef5a33c405a3f85bfcb78303b4c52a"
Sep 10 19:26:28.867624 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.867602    2573 scope.go:117] "RemoveContainer" containerID="b47e711be5d6a92055f2366363167a918531e6e987eaa97271b49b7508b030dd"
Sep 10 19:26:28.876786 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.876758    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"]
Sep 10 19:26:28.879000 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.878979    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-717161-7479d995f7-dpzp5"]
Sep 10 19:26:28.892194 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.892144    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"]
Sep 10 19:26:28.897569 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.897547    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-717161-76879558db-blsnv"]
Sep 10 19:26:28.910064 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.910040    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"]
Sep 10 19:26:28.915674 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:28.915652    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-717161-5d9b8fdcc7-zklbd"]
Sep 10 19:26:30.813639 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:30.813611    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:26:30.833996 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:30.833962    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a3fe555-2097-4a7d-9ba6-5cf5451f39fd" path="/var/lib/kubelet/pods/5a3fe555-2097-4a7d-9ba6-5cf5451f39fd/volumes"
Sep 10 19:26:30.834568 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:30.834547    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833a0591-5dda-4c93-a585-9b7f0bfe2b2e" path="/var/lib/kubelet/pods/833a0591-5dda-4c93-a585-9b7f0bfe2b2e/volumes"
Sep 10 19:26:30.834942 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:30.834925    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a18af411-b51d-43ce-a28f-350acc21fb11" path="/var/lib/kubelet/pods/a18af411-b51d-43ce-a28f-350acc21fb11/volumes"
Sep 10 19:26:35.809945 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:35.809914    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:26:49.218666 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.218633    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"]
Sep 10 19:26:49.222404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.220121    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833a0591-5dda-4c93-a585-9b7f0bfe2b2e" containerName="maas-api"
Sep 10 19:26:49.222404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.220180    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="833a0591-5dda-4c93-a585-9b7f0bfe2b2e" containerName="maas-api"
Sep 10 19:26:49.222404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.220219    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18af411-b51d-43ce-a28f-350acc21fb11" containerName="payload-processing"
Sep 10 19:26:49.222404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.220229    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18af411-b51d-43ce-a28f-350acc21fb11" containerName="payload-processing"
Sep 10 19:26:49.222404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.220254    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a3fe555-2097-4a7d-9ba6-5cf5451f39fd" containerName="payload-pre-processing"
Sep 10 19:26:49.222404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.220265    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a3fe555-2097-4a7d-9ba6-5cf5451f39fd" containerName="payload-pre-processing"
Sep 10 19:26:49.222404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.220465    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="833a0591-5dda-4c93-a585-9b7f0bfe2b2e" containerName="maas-api"
Sep 10 19:26:49.222404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.220490    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18af411-b51d-43ce-a28f-350acc21fb11" containerName="payload-processing"
Sep 10 19:26:49.222404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.220509    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a3fe555-2097-4a7d-9ba6-5cf5451f39fd" containerName="payload-pre-processing"
Sep 10 19:26:49.226033 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.225995    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.230460 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.230431    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-70a2c9\"/\"openshift-service-ca.crt\""
Sep 10 19:26:49.230460 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.230448    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-70a2c9\"/\"noten-model-70a2c9-kserve-self-signed-certs\""
Sep 10 19:26:49.230696 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.230511    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-70a2c9\"/\"default-dockercfg-5rcsb\""
Sep 10 19:26:49.230696 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.230593    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-70a2c9\"/\"kube-root-ca.crt\""
Sep 10 19:26:49.233474 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.233446    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"]
Sep 10 19:26:49.362018 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.361985    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-dshm\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.362018 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.362017    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-home\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.362314 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.362040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-model-cache\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.362314 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.362120    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrx82\" (UniqueName: \"kubernetes.io/projected/1275cf85-53b5-4d6e-91cd-08712d0829e3-kube-api-access-wrx82\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.362314 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.362137    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-tmp-dir\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.362314 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.362179    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1275cf85-53b5-4d6e-91cd-08712d0829e3-tls-certs\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.463594 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.463558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-dshm\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.463594 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.463598    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-home\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.463852 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.463634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-model-cache\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.463852 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.463678    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrx82\" (UniqueName: \"kubernetes.io/projected/1275cf85-53b5-4d6e-91cd-08712d0829e3-kube-api-access-wrx82\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.463852 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.463703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-tmp-dir\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.464034 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.463850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1275cf85-53b5-4d6e-91cd-08712d0829e3-tls-certs\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.464034 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.464003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-home\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.464180 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.464039    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-tmp-dir\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.464180 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.464132    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-model-cache\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.466066 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.466037    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-dshm\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.466564 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.466544    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1275cf85-53b5-4d6e-91cd-08712d0829e3-tls-certs\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.478134 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.478081    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrx82\" (UniqueName: \"kubernetes.io/projected/1275cf85-53b5-4d6e-91cd-08712d0829e3-kube-api-access-wrx82\") pod \"noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") " pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.537130 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.537097    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:26:49.811840 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.811801    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"]
Sep 10 19:26:49.813368 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:26:49.813342    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1275cf85_53b5_4d6e_91cd_08712d0829e3.slice/crio-31d68a1881280f3ec2ccb5547d4c83b028e8a3e71b1064316f413f5b28022df4 WatchSource:0}: Error finding container 31d68a1881280f3ec2ccb5547d4c83b028e8a3e71b1064316f413f5b28022df4: Status 404 returned error can't find the container with id 31d68a1881280f3ec2ccb5547d4c83b028e8a3e71b1064316f413f5b28022df4
Sep 10 19:26:49.961998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.961961    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l" event={"ID":"1275cf85-53b5-4d6e-91cd-08712d0829e3","Type":"ContainerStarted","Data":"7297516ff02ac5ce42d39685bc02576e81bb5956cd432f590eb35eb7e25fcd8d"}
Sep 10 19:26:49.962195 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.962008    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l" event={"ID":"1275cf85-53b5-4d6e-91cd-08712d0829e3","Type":"ContainerStarted","Data":"31d68a1881280f3ec2ccb5547d4c83b028e8a3e71b1064316f413f5b28022df4"}
Sep 10 19:26:49.984087 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:49.984038    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l" podStartSLOduration=0.98402203 podStartE2EDuration="984.02203ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:49.983511539 +0000 UTC m=+945.643308733" watchObservedRunningTime="2026-09-10 19:26:49.98402203 +0000 UTC m=+945.643819234"
Sep 10 19:26:56.834303 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:56.834263    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"]
Sep 10 19:26:59.538053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:26:59.538022    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:27:04.954694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:04.954666    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"]
Sep 10 19:27:04.955177 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:04.954886    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff" podUID="bf704802-21cf-4479-ac5a-efe4e5423849" containerName="maas-api" containerID="cri-o://b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497" gracePeriod=30
Sep 10 19:27:05.016073 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.016032    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"]
Sep 10 19:27:05.016318 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.016278    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn" podUID="59f787fe-af1e-41a2-ba63-a4f7d6b47fba" containerName="payload-pre-processing" containerID="cri-o://bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846" gracePeriod=30
Sep 10 19:27:05.279367 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.279335    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:27:05.306547 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.306516    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:27:05.322649 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.322617    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt8cr\" (UniqueName: \"kubernetes.io/projected/bf704802-21cf-4479-ac5a-efe4e5423849-kube-api-access-rt8cr\") pod \"bf704802-21cf-4479-ac5a-efe4e5423849\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") "
Sep 10 19:27:05.322826 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.322711    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-plugins-config-volume\") pod \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\" (UID: \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\") "
Sep 10 19:27:05.322826 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.322760    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqx6c\" (UniqueName: \"kubernetes.io/projected/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-kube-api-access-nqx6c\") pod \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\" (UID: \"59f787fe-af1e-41a2-ba63-a4f7d6b47fba\") "
Sep 10 19:27:05.322826 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.322813    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf704802-21cf-4479-ac5a-efe4e5423849-openshift-service-ca\") pod \"bf704802-21cf-4479-ac5a-efe4e5423849\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") "
Sep 10 19:27:05.323002 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.322841    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-metrics-certs\") pod \"bf704802-21cf-4479-ac5a-efe4e5423849\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") "
Sep 10 19:27:05.323002 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.322901    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-maas-api-tls\") pod \"bf704802-21cf-4479-ac5a-efe4e5423849\" (UID: \"bf704802-21cf-4479-ac5a-efe4e5423849\") "
Sep 10 19:27:05.323267 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.323230    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59f787fe-af1e-41a2-ba63-a4f7d6b47fba" (UID: "59f787fe-af1e-41a2-ba63-a4f7d6b47fba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:05.323267 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.323252    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf704802-21cf-4479-ac5a-efe4e5423849-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf704802-21cf-4479-ac5a-efe4e5423849" (UID: "bf704802-21cf-4479-ac5a-efe4e5423849"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:05.325949 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.325865    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf704802-21cf-4479-ac5a-efe4e5423849" (UID: "bf704802-21cf-4479-ac5a-efe4e5423849"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:27:05.326327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.326195    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf704802-21cf-4479-ac5a-efe4e5423849-kube-api-access-rt8cr" (OuterVolumeSpecName: "kube-api-access-rt8cr") pod "bf704802-21cf-4479-ac5a-efe4e5423849" (UID: "bf704802-21cf-4479-ac5a-efe4e5423849"). InnerVolumeSpecName "kube-api-access-rt8cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:05.326327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.326237    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf704802-21cf-4479-ac5a-efe4e5423849" (UID: "bf704802-21cf-4479-ac5a-efe4e5423849"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:27:05.326514 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.326491    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-kube-api-access-nqx6c" (OuterVolumeSpecName: "kube-api-access-nqx6c") pod "59f787fe-af1e-41a2-ba63-a4f7d6b47fba" (UID: "59f787fe-af1e-41a2-ba63-a4f7d6b47fba"). InnerVolumeSpecName "kube-api-access-nqx6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:05.424773 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.424727    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:05.424773 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.424763    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqx6c\" (UniqueName: \"kubernetes.io/projected/59f787fe-af1e-41a2-ba63-a4f7d6b47fba-kube-api-access-nqx6c\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:05.424773 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.424774    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf704802-21cf-4479-ac5a-efe4e5423849-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:05.424773 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.424783    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:05.425046 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.424794    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf704802-21cf-4479-ac5a-efe4e5423849-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:05.425046 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:05.424803    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt8cr\" (UniqueName: \"kubernetes.io/projected/bf704802-21cf-4479-ac5a-efe4e5423849-kube-api-access-rt8cr\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:06.053289 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.053232    2573 generic.go:356] "Generic (PLEG): container finished" podID="59f787fe-af1e-41a2-ba63-a4f7d6b47fba" containerID="bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846" exitCode=0
Sep 10 19:27:06.053289 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.053288    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"
Sep 10 19:27:06.053812 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.053322    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn" event={"ID":"59f787fe-af1e-41a2-ba63-a4f7d6b47fba","Type":"ContainerDied","Data":"bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846"}
Sep 10 19:27:06.053812 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.053374    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn" event={"ID":"59f787fe-af1e-41a2-ba63-a4f7d6b47fba","Type":"ContainerDied","Data":"6a63729e580b784f5181e025b09c58df33333ccba220694c959b36332b3fc0f5"}
Sep 10 19:27:06.053812 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.053396    2573 scope.go:117] "RemoveContainer" containerID="bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846"
Sep 10 19:27:06.054976 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.054955    2573 generic.go:356] "Generic (PLEG): container finished" podID="bf704802-21cf-4479-ac5a-efe4e5423849" containerID="b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497" exitCode=0
Sep 10 19:27:06.055037 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.055025    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"
Sep 10 19:27:06.055097 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.055078    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff" event={"ID":"bf704802-21cf-4479-ac5a-efe4e5423849","Type":"ContainerDied","Data":"b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497"}
Sep 10 19:27:06.055194 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.055126    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff" event={"ID":"bf704802-21cf-4479-ac5a-efe4e5423849","Type":"ContainerDied","Data":"42cc83a3b57e6710c466e242845b60751332b4337fc2954f4a288bc002dfdc14"}
Sep 10 19:27:06.064530 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.064503    2573 scope.go:117] "RemoveContainer" containerID="bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846"
Sep 10 19:27:06.064894 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:27:06.064868    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846\": container with ID starting with bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846 not found: ID does not exist" containerID="bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846"
Sep 10 19:27:06.065018 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.064905    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846"} err="failed to get container status \"bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846\": rpc error: code = NotFound desc = could not find container \"bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846\": container with ID starting with bbc2ff77a51fb0d9d8aed23a8c394b26e6cee3d4b5c472711d91bc16f985b846 not found: ID does not exist"
Sep 10 19:27:06.065018 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.064927    2573 scope.go:117] "RemoveContainer" containerID="b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497"
Sep 10 19:27:06.074172 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.074134    2573 scope.go:117] "RemoveContainer" containerID="b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497"
Sep 10 19:27:06.074549 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:27:06.074526    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497\": container with ID starting with b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497 not found: ID does not exist" containerID="b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497"
Sep 10 19:27:06.074611 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.074559    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497"} err="failed to get container status \"b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497\": rpc error: code = NotFound desc = could not find container \"b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497\": container with ID starting with b332b0cab5214196db4adbecd6a98706353038bc5714c2da6d9a8d48c5131497 not found: ID does not exist"
Sep 10 19:27:06.081089 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.081056    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"]
Sep 10 19:27:06.085899 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.085864    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c4487c09-5bd646fc7c-b2cff"]
Sep 10 19:27:06.101975 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.101930    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"]
Sep 10 19:27:06.107104 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.107069    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c4487c09-58f5bcdb66-cvnrn"]
Sep 10 19:27:06.834647 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.834616    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f787fe-af1e-41a2-ba63-a4f7d6b47fba" path="/var/lib/kubelet/pods/59f787fe-af1e-41a2-ba63-a4f7d6b47fba/volumes"
Sep 10 19:27:06.835015 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:06.835002    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf704802-21cf-4479-ac5a-efe4e5423849" path="/var/lib/kubelet/pods/bf704802-21cf-4479-ac5a-efe4e5423849/volumes"
Sep 10 19:27:10.938854 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:10.938817    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"]
Sep 10 19:27:11.045771 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.045728    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"]
Sep 10 19:27:11.046267 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.046251    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf704802-21cf-4479-ac5a-efe4e5423849" containerName="maas-api"
Sep 10 19:27:11.046316 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.046269    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf704802-21cf-4479-ac5a-efe4e5423849" containerName="maas-api"
Sep 10 19:27:11.046316 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.046306    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f787fe-af1e-41a2-ba63-a4f7d6b47fba" containerName="payload-pre-processing"
Sep 10 19:27:11.046316 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.046313    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f787fe-af1e-41a2-ba63-a4f7d6b47fba" containerName="payload-pre-processing"
Sep 10 19:27:11.046414 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.046380    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf704802-21cf-4479-ac5a-efe4e5423849" containerName="maas-api"
Sep 10 19:27:11.046414 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.046389    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f787fe-af1e-41a2-ba63-a4f7d6b47fba" containerName="payload-pre-processing"
Sep 10 19:27:11.051144 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.051122    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"
Sep 10 19:27:11.053788 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.053764    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\""
Sep 10 19:27:11.058216 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.058184    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"]
Sep 10 19:27:11.181562 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.181518    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bdd29-bb96-433d-b349-87a6bdad17fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7\" (UID: \"aa1bdd29-bb96-433d-b349-87a6bdad17fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"
Sep 10 19:27:11.283106 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.283063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bdd29-bb96-433d-b349-87a6bdad17fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7\" (UID: \"aa1bdd29-bb96-433d-b349-87a6bdad17fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"
Sep 10 19:27:11.283754 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.283729    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bdd29-bb96-433d-b349-87a6bdad17fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7\" (UID: \"aa1bdd29-bb96-433d-b349-87a6bdad17fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"
Sep 10 19:27:11.363680 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.363643    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"
Sep 10 19:27:11.645653 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.645618    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"]
Sep 10 19:27:11.647994 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:27:11.647966    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1bdd29_bb96_433d_b349_87a6bdad17fd.slice/crio-55fd774fc3fa72b6e3735a4eb7f6981da7c359e2caff64a5f130ace36a18e734 WatchSource:0}: Error finding container 55fd774fc3fa72b6e3735a4eb7f6981da7c359e2caff64a5f130ace36a18e734: Status 404 returned error can't find the container with id 55fd774fc3fa72b6e3735a4eb7f6981da7c359e2caff64a5f130ace36a18e734
Sep 10 19:27:11.855390 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:11.855348    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l" podUID="1275cf85-53b5-4d6e-91cd-08712d0829e3" containerName="main" containerID="cri-o://7297516ff02ac5ce42d39685bc02576e81bb5956cd432f590eb35eb7e25fcd8d" gracePeriod=45
Sep 10 19:27:12.092291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.092253    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7" event={"ID":"aa1bdd29-bb96-433d-b349-87a6bdad17fd","Type":"ContainerStarted","Data":"55fd774fc3fa72b6e3735a4eb7f6981da7c359e2caff64a5f130ace36a18e734"}
Sep 10 19:27:12.094524 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.094488    2573 generic.go:356] "Generic (PLEG): container finished" podID="1275cf85-53b5-4d6e-91cd-08712d0829e3" containerID="7297516ff02ac5ce42d39685bc02576e81bb5956cd432f590eb35eb7e25fcd8d" exitCode=0
Sep 10 19:27:12.094677 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.094519    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l" event={"ID":"1275cf85-53b5-4d6e-91cd-08712d0829e3","Type":"ContainerDied","Data":"7297516ff02ac5ce42d39685bc02576e81bb5956cd432f590eb35eb7e25fcd8d"}
Sep 10 19:27:12.120386 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.120353    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:27:12.192014 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.191982    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1275cf85-53b5-4d6e-91cd-08712d0829e3-tls-certs\") pod \"1275cf85-53b5-4d6e-91cd-08712d0829e3\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") "
Sep 10 19:27:12.192236 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192046    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-dshm\") pod \"1275cf85-53b5-4d6e-91cd-08712d0829e3\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") "
Sep 10 19:27:12.192236 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192073    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-tmp-dir\") pod \"1275cf85-53b5-4d6e-91cd-08712d0829e3\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") "
Sep 10 19:27:12.192236 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192091    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrx82\" (UniqueName: \"kubernetes.io/projected/1275cf85-53b5-4d6e-91cd-08712d0829e3-kube-api-access-wrx82\") pod \"1275cf85-53b5-4d6e-91cd-08712d0829e3\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") "
Sep 10 19:27:12.192236 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192123    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-model-cache\") pod \"1275cf85-53b5-4d6e-91cd-08712d0829e3\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") "
Sep 10 19:27:12.192469 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192246    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-home\") pod \"1275cf85-53b5-4d6e-91cd-08712d0829e3\" (UID: \"1275cf85-53b5-4d6e-91cd-08712d0829e3\") "
Sep 10 19:27:12.192469 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192445    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1275cf85-53b5-4d6e-91cd-08712d0829e3" (UID: "1275cf85-53b5-4d6e-91cd-08712d0829e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:12.192589 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192547    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "1275cf85-53b5-4d6e-91cd-08712d0829e3" (UID: "1275cf85-53b5-4d6e-91cd-08712d0829e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:12.192686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192663    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-home" (OuterVolumeSpecName: "home") pod "1275cf85-53b5-4d6e-91cd-08712d0829e3" (UID: "1275cf85-53b5-4d6e-91cd-08712d0829e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:12.192758 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192741    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:12.192980 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.192761    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:12.194677 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.194638    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1275cf85-53b5-4d6e-91cd-08712d0829e3-kube-api-access-wrx82" (OuterVolumeSpecName: "kube-api-access-wrx82") pod "1275cf85-53b5-4d6e-91cd-08712d0829e3" (UID: "1275cf85-53b5-4d6e-91cd-08712d0829e3"). InnerVolumeSpecName "kube-api-access-wrx82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:12.194677 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.194654    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1275cf85-53b5-4d6e-91cd-08712d0829e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1275cf85-53b5-4d6e-91cd-08712d0829e3" (UID: "1275cf85-53b5-4d6e-91cd-08712d0829e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:27:12.194851 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.194708    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-dshm" (OuterVolumeSpecName: "dshm") pod "1275cf85-53b5-4d6e-91cd-08712d0829e3" (UID: "1275cf85-53b5-4d6e-91cd-08712d0829e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:12.294300 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.294267    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrx82\" (UniqueName: \"kubernetes.io/projected/1275cf85-53b5-4d6e-91cd-08712d0829e3-kube-api-access-wrx82\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:12.294300 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.294296    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:12.294300 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.294305    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1275cf85-53b5-4d6e-91cd-08712d0829e3-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:12.294548 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:12.294314    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1275cf85-53b5-4d6e-91cd-08712d0829e3-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:13.103458 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:13.103372    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l" event={"ID":"1275cf85-53b5-4d6e-91cd-08712d0829e3","Type":"ContainerDied","Data":"31d68a1881280f3ec2ccb5547d4c83b028e8a3e71b1064316f413f5b28022df4"}
Sep 10 19:27:13.103458 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:13.103436    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"
Sep 10 19:27:13.103458 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:13.103448    2573 scope.go:117] "RemoveContainer" containerID="7297516ff02ac5ce42d39685bc02576e81bb5956cd432f590eb35eb7e25fcd8d"
Sep 10 19:27:13.126588 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:13.126545    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"]
Sep 10 19:27:13.131764 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:13.131726    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-70a2c9/noten-model-70a2c9-kserve-84bf8f6d8c-kjd7l"]
Sep 10 19:27:14.111692 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:14.111654    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7" event={"ID":"aa1bdd29-bb96-433d-b349-87a6bdad17fd","Type":"ContainerStarted","Data":"b90b1edc8d5cf1a30e49ec44a442d2639f617fe45dc791b1fd25dcd9c2274ade"}
Sep 10 19:27:14.132112 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:14.132043    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7" podStartSLOduration=1.070143832 podStartE2EDuration="3.132028013s" totalImagesPullingTime="2.061884181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:14.129506369 +0000 UTC m=+969.789303562" watchObservedRunningTime="2026-09-10 19:27:14.132028013 +0000 UTC m=+969.791825211"
Sep 10 19:27:14.834470 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:14.834431    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1275cf85-53b5-4d6e-91cd-08712d0829e3" path="/var/lib/kubelet/pods/1275cf85-53b5-4d6e-91cd-08712d0829e3/volumes"
Sep 10 19:27:15.118888 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:15.118792    2573 generic.go:356] "Generic (PLEG): container finished" podID="aa1bdd29-bb96-433d-b349-87a6bdad17fd" containerID="b90b1edc8d5cf1a30e49ec44a442d2639f617fe45dc791b1fd25dcd9c2274ade" exitCode=0
Sep 10 19:27:15.118888 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:15.118845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7" event={"ID":"aa1bdd29-bb96-433d-b349-87a6bdad17fd","Type":"ContainerDied","Data":"b90b1edc8d5cf1a30e49ec44a442d2639f617fe45dc791b1fd25dcd9c2274ade"}
Sep 10 19:27:16.261645 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:16.261615    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"
Sep 10 19:27:16.337546 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:16.337510    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bdd29-bb96-433d-b349-87a6bdad17fd-maas-api-service-ca\") pod \"aa1bdd29-bb96-433d-b349-87a6bdad17fd\" (UID: \"aa1bdd29-bb96-433d-b349-87a6bdad17fd\") "
Sep 10 19:27:16.337899 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:16.337872    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1bdd29-bb96-433d-b349-87a6bdad17fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa1bdd29-bb96-433d-b349-87a6bdad17fd" (UID: "aa1bdd29-bb96-433d-b349-87a6bdad17fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:16.438537 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:16.438452    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa1bdd29-bb96-433d-b349-87a6bdad17fd-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:17.131145 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:17.131049    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7" event={"ID":"aa1bdd29-bb96-433d-b349-87a6bdad17fd","Type":"ContainerDied","Data":"55fd774fc3fa72b6e3735a4eb7f6981da7c359e2caff64a5f130ace36a18e734"}
Sep 10 19:27:17.131145 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:17.131084    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"
Sep 10 19:27:17.131145 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:17.131112    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55fd774fc3fa72b6e3735a4eb7f6981da7c359e2caff64a5f130ace36a18e734"
Sep 10 19:27:22.097207 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:22.097144    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"]
Sep 10 19:27:22.099768 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:22.099734    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6b562c-rh2m7"]
Sep 10 19:27:22.834423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:22.834381    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa1bdd29-bb96-433d-b349-87a6bdad17fd" path="/var/lib/kubelet/pods/aa1bdd29-bb96-433d-b349-87a6bdad17fd/volumes"
Sep 10 19:27:25.959470 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:25.959368    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv" podUID="33d274f0-bda5-4bdd-8231-6fa19c481910" containerName="main" containerID="cri-o://479899936b66cc06c3dd30c3adfec8028709ec22178b9d36891f76b1973af05c" gracePeriod=45
Sep 10 19:27:26.182883 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.182846    2573 generic.go:356] "Generic (PLEG): container finished" podID="33d274f0-bda5-4bdd-8231-6fa19c481910" containerID="479899936b66cc06c3dd30c3adfec8028709ec22178b9d36891f76b1973af05c" exitCode=0
Sep 10 19:27:26.183078 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.182916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv" event={"ID":"33d274f0-bda5-4bdd-8231-6fa19c481910","Type":"ContainerDied","Data":"479899936b66cc06c3dd30c3adfec8028709ec22178b9d36891f76b1973af05c"}
Sep 10 19:27:26.219755 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.219690    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:27:26.332972 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.332932    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-model-cache\") pod \"33d274f0-bda5-4bdd-8231-6fa19c481910\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") "
Sep 10 19:27:26.333189 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333004    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-home\") pod \"33d274f0-bda5-4bdd-8231-6fa19c481910\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") "
Sep 10 19:27:26.333189 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333069    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d274f0-bda5-4bdd-8231-6fa19c481910-tls-certs\") pod \"33d274f0-bda5-4bdd-8231-6fa19c481910\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") "
Sep 10 19:27:26.333189 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333120    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-tmp-dir\") pod \"33d274f0-bda5-4bdd-8231-6fa19c481910\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") "
Sep 10 19:27:26.333189 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333172    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf2zl\" (UniqueName: \"kubernetes.io/projected/33d274f0-bda5-4bdd-8231-6fa19c481910-kube-api-access-pf2zl\") pod \"33d274f0-bda5-4bdd-8231-6fa19c481910\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") "
Sep 10 19:27:26.333423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333197    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-model-cache" (OuterVolumeSpecName: "model-cache") pod "33d274f0-bda5-4bdd-8231-6fa19c481910" (UID: "33d274f0-bda5-4bdd-8231-6fa19c481910"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:26.333423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333285    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-home" (OuterVolumeSpecName: "home") pod "33d274f0-bda5-4bdd-8231-6fa19c481910" (UID: "33d274f0-bda5-4bdd-8231-6fa19c481910"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:26.333423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333295    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-dshm\") pod \"33d274f0-bda5-4bdd-8231-6fa19c481910\" (UID: \"33d274f0-bda5-4bdd-8231-6fa19c481910\") "
Sep 10 19:27:26.333423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333387    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33d274f0-bda5-4bdd-8231-6fa19c481910" (UID: "33d274f0-bda5-4bdd-8231-6fa19c481910"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:26.333677 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333656    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:26.333740 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333682    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:26.333740 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.333698    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:26.335468 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.335444    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d274f0-bda5-4bdd-8231-6fa19c481910-kube-api-access-pf2zl" (OuterVolumeSpecName: "kube-api-access-pf2zl") pod "33d274f0-bda5-4bdd-8231-6fa19c481910" (UID: "33d274f0-bda5-4bdd-8231-6fa19c481910"). InnerVolumeSpecName "kube-api-access-pf2zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:26.335839 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.335804    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-dshm" (OuterVolumeSpecName: "dshm") pod "33d274f0-bda5-4bdd-8231-6fa19c481910" (UID: "33d274f0-bda5-4bdd-8231-6fa19c481910"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:26.335929 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.335823    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d274f0-bda5-4bdd-8231-6fa19c481910-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33d274f0-bda5-4bdd-8231-6fa19c481910" (UID: "33d274f0-bda5-4bdd-8231-6fa19c481910"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:27:26.434812 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.434757    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d274f0-bda5-4bdd-8231-6fa19c481910-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:26.434812 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.434807    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf2zl\" (UniqueName: \"kubernetes.io/projected/33d274f0-bda5-4bdd-8231-6fa19c481910-kube-api-access-pf2zl\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:26.434812 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:26.434816    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d274f0-bda5-4bdd-8231-6fa19c481910-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:27.190459 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:27.190360    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"
Sep 10 19:27:27.190459 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:27.190380    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv" event={"ID":"33d274f0-bda5-4bdd-8231-6fa19c481910","Type":"ContainerDied","Data":"fdee38bd2b7d676866107e7f527b1a63750c5b1a02dfe5c39112adf096da83ed"}
Sep 10 19:27:27.190459 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:27.190458    2573 scope.go:117] "RemoveContainer" containerID="479899936b66cc06c3dd30c3adfec8028709ec22178b9d36891f76b1973af05c"
Sep 10 19:27:27.208278 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:27.208247    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"]
Sep 10 19:27:27.212082 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:27.212054    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b960a2/test-model-b960a2-kserve-57884d8585-pfrjv"]
Sep 10 19:27:28.833865 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:28.833829    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d274f0-bda5-4bdd-8231-6fa19c481910" path="/var/lib/kubelet/pods/33d274f0-bda5-4bdd-8231-6fa19c481910/volumes"
Sep 10 19:27:30.612178 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612130    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"]
Sep 10 19:27:30.612597 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612583    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d274f0-bda5-4bdd-8231-6fa19c481910" containerName="main"
Sep 10 19:27:30.612597 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612598    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d274f0-bda5-4bdd-8231-6fa19c481910" containerName="main"
Sep 10 19:27:30.612682 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612612    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1275cf85-53b5-4d6e-91cd-08712d0829e3" containerName="main"
Sep 10 19:27:30.612682 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612617    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1275cf85-53b5-4d6e-91cd-08712d0829e3" containerName="main"
Sep 10 19:27:30.612682 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612634    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa1bdd29-bb96-433d-b349-87a6bdad17fd" containerName="revoke-keys"
Sep 10 19:27:30.612682 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612641    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa1bdd29-bb96-433d-b349-87a6bdad17fd" containerName="revoke-keys"
Sep 10 19:27:30.612825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612711    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1275cf85-53b5-4d6e-91cd-08712d0829e3" containerName="main"
Sep 10 19:27:30.612825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612725    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa1bdd29-bb96-433d-b349-87a6bdad17fd" containerName="revoke-keys"
Sep 10 19:27:30.612825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.612734    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d274f0-bda5-4bdd-8231-6fa19c481910" containerName="main"
Sep 10 19:27:30.616009 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.615991    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"
Sep 10 19:27:30.619685 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.619658    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\""
Sep 10 19:27:30.631193 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.631159    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"]
Sep 10 19:27:30.776865 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.776829    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b7a2d0-6bde-4f4f-a005-b6649cdf6938-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z\" (UID: \"b6b7a2d0-6bde-4f4f-a005-b6649cdf6938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"
Sep 10 19:27:30.878437 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.878341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b7a2d0-6bde-4f4f-a005-b6649cdf6938-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z\" (UID: \"b6b7a2d0-6bde-4f4f-a005-b6649cdf6938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"
Sep 10 19:27:30.878982 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.878961    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b7a2d0-6bde-4f4f-a005-b6649cdf6938-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z\" (UID: \"b6b7a2d0-6bde-4f4f-a005-b6649cdf6938\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"
Sep 10 19:27:30.926371 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:30.926334    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"
Sep 10 19:27:31.212092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:31.212059    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"]
Sep 10 19:27:31.214167 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:27:31.214120    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b7a2d0_6bde_4f4f_a005_b6649cdf6938.slice/crio-fe4bc37d90e7736ae4d25c33f73fbe73f5e766722b1933023fe4ce66b967bb0a WatchSource:0}: Error finding container fe4bc37d90e7736ae4d25c33f73fbe73f5e766722b1933023fe4ce66b967bb0a: Status 404 returned error can't find the container with id fe4bc37d90e7736ae4d25c33f73fbe73f5e766722b1933023fe4ce66b967bb0a
Sep 10 19:27:32.226659 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.226560    2573 generic.go:356] "Generic (PLEG): container finished" podID="b6b7a2d0-6bde-4f4f-a005-b6649cdf6938" containerID="224de71471ebc7eb7e52a784d92c89455d7798a6df16a705e1fcab4f01123469" exitCode=0
Sep 10 19:27:32.226659 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.226646    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z" event={"ID":"b6b7a2d0-6bde-4f4f-a005-b6649cdf6938","Type":"ContainerDied","Data":"224de71471ebc7eb7e52a784d92c89455d7798a6df16a705e1fcab4f01123469"}
Sep 10 19:27:32.227111 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.226741    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z" event={"ID":"b6b7a2d0-6bde-4f4f-a005-b6649cdf6938","Type":"ContainerStarted","Data":"fe4bc37d90e7736ae4d25c33f73fbe73f5e766722b1933023fe4ce66b967bb0a"}
Sep 10 19:27:32.625117 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.625078    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"]
Sep 10 19:27:32.628798 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.628773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.631735 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.631712    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-cfa578-serving-cert\""
Sep 10 19:27:32.632765 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.632746    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 19:27:32.632889 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.632786    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lw7bv\""
Sep 10 19:27:32.655119 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.655079    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"]
Sep 10 19:27:32.797365 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.797324    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbv2n\" (UniqueName: \"kubernetes.io/projected/63d35b8f-38f0-447b-b60e-2108d67ceb40-kube-api-access-vbv2n\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.797565 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.797485    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d35b8f-38f0-447b-b60e-2108d67ceb40-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.797565 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.797531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.797662 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.797565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.898615 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.898515    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d35b8f-38f0-447b-b60e-2108d67ceb40-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.898615 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.898557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.898615 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.898596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.898905 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.898644    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbv2n\" (UniqueName: \"kubernetes.io/projected/63d35b8f-38f0-447b-b60e-2108d67ceb40-kube-api-access-vbv2n\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.899338 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.899309    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d35b8f-38f0-447b-b60e-2108d67ceb40-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.901252 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.901230    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.901354 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.901276    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.911995 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.911969    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbv2n\" (UniqueName: \"kubernetes.io/projected/63d35b8f-38f0-447b-b60e-2108d67ceb40-kube-api-access-vbv2n\") pod \"maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:32.940739 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:32.940703    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:33.217913 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.217830    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"]
Sep 10 19:27:33.222216 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:27:33.222185    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d35b8f_38f0_447b_b60e_2108d67ceb40.slice/crio-a374ca3ac9c3371ae218ad20d77cd8a7cdd02f3b715869c77c334f8eae1fafc1 WatchSource:0}: Error finding container a374ca3ac9c3371ae218ad20d77cd8a7cdd02f3b715869c77c334f8eae1fafc1: Status 404 returned error can't find the container with id a374ca3ac9c3371ae218ad20d77cd8a7cdd02f3b715869c77c334f8eae1fafc1
Sep 10 19:27:33.234901 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.234865    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd" event={"ID":"63d35b8f-38f0-447b-b60e-2108d67ceb40","Type":"ContainerStarted","Data":"a374ca3ac9c3371ae218ad20d77cd8a7cdd02f3b715869c77c334f8eae1fafc1"}
Sep 10 19:27:33.352727 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.352700    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"]
Sep 10 19:27:33.357586 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.357562    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:33.360293 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.360268    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"
Sep 10 19:27:33.361407 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.361391    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-cfa578-dockercfg-zdhbj\""
Sep 10 19:27:33.362164 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.362130    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-cfa578\""
Sep 10 19:27:33.367010 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.366986    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"]
Sep 10 19:27:33.506072 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.505965    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b7a2d0-6bde-4f4f-a005-b6649cdf6938-maas-api-service-ca\") pod \"b6b7a2d0-6bde-4f4f-a005-b6649cdf6938\" (UID: \"b6b7a2d0-6bde-4f4f-a005-b6649cdf6938\") "
Sep 10 19:27:33.506301 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.506213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz22n\" (UniqueName: \"kubernetes.io/projected/0156c21b-afca-4717-920f-62ffff92a259-kube-api-access-sz22n\") pod \"payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q\" (UID: \"0156c21b-afca-4717-920f-62ffff92a259\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:33.506407 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.506383    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b7a2d0-6bde-4f4f-a005-b6649cdf6938-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6b7a2d0-6bde-4f4f-a005-b6649cdf6938" (UID: "b6b7a2d0-6bde-4f4f-a005-b6649cdf6938"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:33.506451 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.506401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0156c21b-afca-4717-920f-62ffff92a259-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q\" (UID: \"0156c21b-afca-4717-920f-62ffff92a259\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:33.506501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.506483    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b7a2d0-6bde-4f4f-a005-b6649cdf6938-maas-api-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:33.606969 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.606942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0156c21b-afca-4717-920f-62ffff92a259-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q\" (UID: \"0156c21b-afca-4717-920f-62ffff92a259\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:33.607099 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.606993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz22n\" (UniqueName: \"kubernetes.io/projected/0156c21b-afca-4717-920f-62ffff92a259-kube-api-access-sz22n\") pod \"payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q\" (UID: \"0156c21b-afca-4717-920f-62ffff92a259\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:33.607733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.607706    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0156c21b-afca-4717-920f-62ffff92a259-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q\" (UID: \"0156c21b-afca-4717-920f-62ffff92a259\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:33.625222 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.625186    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz22n\" (UniqueName: \"kubernetes.io/projected/0156c21b-afca-4717-920f-62ffff92a259-kube-api-access-sz22n\") pod \"payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q\" (UID: \"0156c21b-afca-4717-920f-62ffff92a259\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:33.669577 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.669536    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:33.947213 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:33.946974    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"]
Sep 10 19:27:33.949938 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:27:33.949909    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0156c21b_afca_4717_920f_62ffff92a259.slice/crio-99d31d79ebdbcbc0dc62b0886a24fcd63053566aedd8db799f118efdb5ba9efb WatchSource:0}: Error finding container 99d31d79ebdbcbc0dc62b0886a24fcd63053566aedd8db799f118efdb5ba9efb: Status 404 returned error can't find the container with id 99d31d79ebdbcbc0dc62b0886a24fcd63053566aedd8db799f118efdb5ba9efb
Sep 10 19:27:34.242765 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.242666    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z" event={"ID":"b6b7a2d0-6bde-4f4f-a005-b6649cdf6938","Type":"ContainerDied","Data":"fe4bc37d90e7736ae4d25c33f73fbe73f5e766722b1933023fe4ce66b967bb0a"}
Sep 10 19:27:34.242765 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.242718    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe4bc37d90e7736ae4d25c33f73fbe73f5e766722b1933023fe4ce66b967bb0a"
Sep 10 19:27:34.242765 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.242674    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"
Sep 10 19:27:34.244556 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.244523    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd" event={"ID":"63d35b8f-38f0-447b-b60e-2108d67ceb40","Type":"ContainerStarted","Data":"26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188"}
Sep 10 19:27:34.244704 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.244597    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:34.246382 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.246358    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q" event={"ID":"0156c21b-afca-4717-920f-62ffff92a259","Type":"ContainerStarted","Data":"ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d"}
Sep 10 19:27:34.246492 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.246404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q" event={"ID":"0156c21b-afca-4717-920f-62ffff92a259","Type":"ContainerStarted","Data":"99d31d79ebdbcbc0dc62b0886a24fcd63053566aedd8db799f118efdb5ba9efb"}
Sep 10 19:27:34.246492 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.246442    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:34.269783 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.269709    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd" podStartSLOduration=1.897273197 podStartE2EDuration="2.269685927s" totalImagesPullingTime="372.41273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:34.268460825 +0000 UTC m=+989.928258018" watchObservedRunningTime="2026-09-10 19:27:34.269685927 +0000 UTC m=+989.929483125"
Sep 10 19:27:34.310632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:34.310575    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q" podStartSLOduration=1.310554302 podStartE2EDuration="1.310554302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:34.308413084 +0000 UTC m=+989.968210279" watchObservedRunningTime="2026-09-10 19:27:34.310554302 +0000 UTC m=+989.970351495"
Sep 10 19:27:40.258740 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:40.258707    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:27:41.889353 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:41.889317    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"]
Sep 10 19:27:41.889772 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:41.889602    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz" podUID="f34fbfff-47c6-409d-afe6-8770428d7219" containerName="payload-pre-processing" containerID="cri-o://244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c" gracePeriod=30
Sep 10 19:27:42.161277 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.161245    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:27:42.189959 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.189917    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slkd9\" (UniqueName: \"kubernetes.io/projected/f34fbfff-47c6-409d-afe6-8770428d7219-kube-api-access-slkd9\") pod \"f34fbfff-47c6-409d-afe6-8770428d7219\" (UID: \"f34fbfff-47c6-409d-afe6-8770428d7219\") "
Sep 10 19:27:42.190240 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.190027    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34fbfff-47c6-409d-afe6-8770428d7219-plugins-config-volume\") pod \"f34fbfff-47c6-409d-afe6-8770428d7219\" (UID: \"f34fbfff-47c6-409d-afe6-8770428d7219\") "
Sep 10 19:27:42.190436 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.190408    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34fbfff-47c6-409d-afe6-8770428d7219-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f34fbfff-47c6-409d-afe6-8770428d7219" (UID: "f34fbfff-47c6-409d-afe6-8770428d7219"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:42.192312 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.192279    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34fbfff-47c6-409d-afe6-8770428d7219-kube-api-access-slkd9" (OuterVolumeSpecName: "kube-api-access-slkd9") pod "f34fbfff-47c6-409d-afe6-8770428d7219" (UID: "f34fbfff-47c6-409d-afe6-8770428d7219"). InnerVolumeSpecName "kube-api-access-slkd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:42.215965 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.215923    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"]
Sep 10 19:27:42.231881 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.231846    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b960a2-7jn7z"]
Sep 10 19:27:42.291019 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.290984    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slkd9\" (UniqueName: \"kubernetes.io/projected/f34fbfff-47c6-409d-afe6-8770428d7219-kube-api-access-slkd9\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:42.291019 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.291019    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f34fbfff-47c6-409d-afe6-8770428d7219-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:27:42.291674 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.291653    2573 generic.go:356] "Generic (PLEG): container finished" podID="f34fbfff-47c6-409d-afe6-8770428d7219" containerID="244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c" exitCode=0
Sep 10 19:27:42.291753 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.291710    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz" event={"ID":"f34fbfff-47c6-409d-afe6-8770428d7219","Type":"ContainerDied","Data":"244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c"}
Sep 10 19:27:42.291753 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.291711    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"
Sep 10 19:27:42.291753 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.291749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz" event={"ID":"f34fbfff-47c6-409d-afe6-8770428d7219","Type":"ContainerDied","Data":"e697591971b45a35c411300023a102946aa170ad697844f58372d4c90f01ec4c"}
Sep 10 19:27:42.291909 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.291770    2573 scope.go:117] "RemoveContainer" containerID="244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c"
Sep 10 19:27:42.301291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.301268    2573 scope.go:117] "RemoveContainer" containerID="244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c"
Sep 10 19:27:42.301651 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:27:42.301631    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c\": container with ID starting with 244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c not found: ID does not exist" containerID="244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c"
Sep 10 19:27:42.301730 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.301664    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c"} err="failed to get container status \"244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c\": rpc error: code = NotFound desc = could not find container \"244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c\": container with ID starting with 244953fb4f85a2db6563424a39390a27da5899e055257135abb1635d0ba9b28c not found: ID does not exist"
Sep 10 19:27:42.326199 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.326143    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"]
Sep 10 19:27:42.331077 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.331045    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b960a2-69c596748b-rdrxz"]
Sep 10 19:27:42.833852 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.833817    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6b7a2d0-6bde-4f4f-a005-b6649cdf6938" path="/var/lib/kubelet/pods/b6b7a2d0-6bde-4f4f-a005-b6649cdf6938/volumes"
Sep 10 19:27:42.834261 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:42.834243    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f34fbfff-47c6-409d-afe6-8770428d7219" path="/var/lib/kubelet/pods/f34fbfff-47c6-409d-afe6-8770428d7219/volumes"
Sep 10 19:27:45.255217 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:45.255187    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:27:55.772931 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.772893    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"]
Sep 10 19:27:55.773440 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.773360    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f34fbfff-47c6-409d-afe6-8770428d7219" containerName="payload-pre-processing"
Sep 10 19:27:55.773440 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.773375    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f34fbfff-47c6-409d-afe6-8770428d7219" containerName="payload-pre-processing"
Sep 10 19:27:55.773440 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.773400    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b7a2d0-6bde-4f4f-a005-b6649cdf6938" containerName="revoke-keys"
Sep 10 19:27:55.773440 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.773405    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b7a2d0-6bde-4f4f-a005-b6649cdf6938" containerName="revoke-keys"
Sep 10 19:27:55.773621 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.773469    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f34fbfff-47c6-409d-afe6-8770428d7219" containerName="payload-pre-processing"
Sep 10 19:27:55.773621 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.773480    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6b7a2d0-6bde-4f4f-a005-b6649cdf6938" containerName="revoke-keys"
Sep 10 19:27:55.780371 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.780345    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:27:55.783335 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.783304    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b64b1c\""
Sep 10 19:27:55.783448 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.783359    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b64b1c-dockercfg-j7vpm\""
Sep 10 19:27:55.789335 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.789310    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"]
Sep 10 19:27:55.821039 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.820997    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d42752-7e3e-4c22-b53c-be9928ec9b95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4\" (UID: \"88d42752-7e3e-4c22-b53c-be9928ec9b95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:27:55.821252 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.821077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/88d42752-7e3e-4c22-b53c-be9928ec9b95-kube-api-access-h74j4\") pod \"payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4\" (UID: \"88d42752-7e3e-4c22-b53c-be9928ec9b95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:27:55.922036 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.921990    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d42752-7e3e-4c22-b53c-be9928ec9b95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4\" (UID: \"88d42752-7e3e-4c22-b53c-be9928ec9b95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:27:55.922282 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.922077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/88d42752-7e3e-4c22-b53c-be9928ec9b95-kube-api-access-h74j4\") pod \"payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4\" (UID: \"88d42752-7e3e-4c22-b53c-be9928ec9b95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:27:55.922865 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.922843    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d42752-7e3e-4c22-b53c-be9928ec9b95-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4\" (UID: \"88d42752-7e3e-4c22-b53c-be9928ec9b95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:27:55.931654 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:55.931633    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/88d42752-7e3e-4c22-b53c-be9928ec9b95-kube-api-access-h74j4\") pod \"payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4\" (UID: \"88d42752-7e3e-4c22-b53c-be9928ec9b95\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:27:56.092072 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:56.091977    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:27:56.372336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:56.372311    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"]
Sep 10 19:27:56.374123 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:27:56.374089    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d42752_7e3e_4c22_b53c_be9928ec9b95.slice/crio-6eb88406491faa97077b0645f24ad98df0e519dad259ccc5c46e65aa1e275004 WatchSource:0}: Error finding container 6eb88406491faa97077b0645f24ad98df0e519dad259ccc5c46e65aa1e275004: Status 404 returned error can't find the container with id 6eb88406491faa97077b0645f24ad98df0e519dad259ccc5c46e65aa1e275004
Sep 10 19:27:57.381757 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:57.381720    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4" event={"ID":"88d42752-7e3e-4c22-b53c-be9928ec9b95","Type":"ContainerStarted","Data":"f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a"}
Sep 10 19:27:57.382217 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:57.381775    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4" event={"ID":"88d42752-7e3e-4c22-b53c-be9928ec9b95","Type":"ContainerStarted","Data":"6eb88406491faa97077b0645f24ad98df0e519dad259ccc5c46e65aa1e275004"}
Sep 10 19:27:57.382217 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:57.381833    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:27:57.401249 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:27:57.401192    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4" podStartSLOduration=2.401174631 podStartE2EDuration="2.401174631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:57.399825605 +0000 UTC m=+1013.059622812" watchObservedRunningTime="2026-09-10 19:27:57.401174631 +0000 UTC m=+1013.060971818"
Sep 10 19:28:01.462631 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.462597    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"]
Sep 10 19:28:01.466536 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.466508    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.469913 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.469889    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-94ca92-serving-cert\""
Sep 10 19:28:01.487537 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.487498    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"]
Sep 10 19:28:01.579429 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.579391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.579619 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.579477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2ws\" (UniqueName: \"kubernetes.io/projected/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-kube-api-access-zn2ws\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.579619 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.579502    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.579619 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.579523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.680455 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.680410    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.680661 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.680495    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2ws\" (UniqueName: \"kubernetes.io/projected/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-kube-api-access-zn2ws\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.680661 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.680518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.680661 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.680538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.680837 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:28:01.680663    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-serving-cert: secret "maas-api-e2e-shared-b-w2-94ca92-serving-cert" not found
Sep 10 19:28:01.680837 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:28:01.680733    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-maas-api-tls podName:2795cad8-c5d4-4694-8a22-0a17a2e2c73e nodeName:}" failed. No retries permitted until 2026-09-10 19:28:02.180715508 +0000 UTC m=+1017.840512713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-maas-api-tls") pod "maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb" (UID: "2795cad8-c5d4-4694-8a22-0a17a2e2c73e") : secret "maas-api-e2e-shared-b-w2-94ca92-serving-cert" not found
Sep 10 19:28:01.681274 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.681254    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.683035 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.683015    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:01.754396 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:01.754319    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2ws\" (UniqueName: \"kubernetes.io/projected/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-kube-api-access-zn2ws\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:02.185358 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:02.185298    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:02.188011 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:02.187981    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:02.378265 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:02.378227    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:02.672624 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:02.672597    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"]
Sep 10 19:28:02.674104 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:28:02.674068    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2795cad8_c5d4_4694_8a22_0a17a2e2c73e.slice/crio-176523d1da7e13b825029e674efcbfce8f08f652e378086b10a81d04dad0c6ec WatchSource:0}: Error finding container 176523d1da7e13b825029e674efcbfce8f08f652e378086b10a81d04dad0c6ec: Status 404 returned error can't find the container with id 176523d1da7e13b825029e674efcbfce8f08f652e378086b10a81d04dad0c6ec
Sep 10 19:28:02.675611 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:02.675587    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:28:03.272509 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.272471    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"]
Sep 10 19:28:03.276142 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.276122    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:03.278956 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.278921    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-94ca92-dockercfg-mfls7\""
Sep 10 19:28:03.279346 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.279329    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-94ca92\""
Sep 10 19:28:03.300505 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.300412    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"]
Sep 10 19:28:03.397396 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.397353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfvm\" (UniqueName: \"kubernetes.io/projected/0a190b42-a310-43a2-a93d-6f95a298f19c-kube-api-access-nhfvm\") pod \"payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7\" (UID: \"0a190b42-a310-43a2-a93d-6f95a298f19c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:03.397396 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.397395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a190b42-a310-43a2-a93d-6f95a298f19c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7\" (UID: \"0a190b42-a310-43a2-a93d-6f95a298f19c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:03.420423 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.420384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb" event={"ID":"2795cad8-c5d4-4694-8a22-0a17a2e2c73e","Type":"ContainerStarted","Data":"819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8"}
Sep 10 19:28:03.420632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.420438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb" event={"ID":"2795cad8-c5d4-4694-8a22-0a17a2e2c73e","Type":"ContainerStarted","Data":"176523d1da7e13b825029e674efcbfce8f08f652e378086b10a81d04dad0c6ec"}
Sep 10 19:28:03.420632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.420527    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:03.478977 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.478894    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb" podStartSLOduration=2.106276937 podStartE2EDuration="2.478874777s" totalImagesPullingTime="372.59784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:03.470343156 +0000 UTC m=+1019.130140348" watchObservedRunningTime="2026-09-10 19:28:03.478874777 +0000 UTC m=+1019.138671976"
Sep 10 19:28:03.499183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.499108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfvm\" (UniqueName: \"kubernetes.io/projected/0a190b42-a310-43a2-a93d-6f95a298f19c-kube-api-access-nhfvm\") pod \"payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7\" (UID: \"0a190b42-a310-43a2-a93d-6f95a298f19c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:03.499389 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.499198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a190b42-a310-43a2-a93d-6f95a298f19c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7\" (UID: \"0a190b42-a310-43a2-a93d-6f95a298f19c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:03.499939 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.499916    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a190b42-a310-43a2-a93d-6f95a298f19c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7\" (UID: \"0a190b42-a310-43a2-a93d-6f95a298f19c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:03.533982 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.533948    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfvm\" (UniqueName: \"kubernetes.io/projected/0a190b42-a310-43a2-a93d-6f95a298f19c-kube-api-access-nhfvm\") pod \"payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7\" (UID: \"0a190b42-a310-43a2-a93d-6f95a298f19c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:03.586991 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.586894    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:03.868795 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:03.868768    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"]
Sep 10 19:28:03.871252 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:28:03.871222    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a190b42_a310_43a2_a93d_6f95a298f19c.slice/crio-ae12054180de3e8eab4f3f4e006e317d4d1676e2f91b57e2c0f223a868343633 WatchSource:0}: Error finding container ae12054180de3e8eab4f3f4e006e317d4d1676e2f91b57e2c0f223a868343633: Status 404 returned error can't find the container with id ae12054180de3e8eab4f3f4e006e317d4d1676e2f91b57e2c0f223a868343633
Sep 10 19:28:04.434770 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:04.434725    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7" event={"ID":"0a190b42-a310-43a2-a93d-6f95a298f19c","Type":"ContainerStarted","Data":"d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26"}
Sep 10 19:28:04.434988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:04.434921    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:04.434988 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:04.434967    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7" event={"ID":"0a190b42-a310-43a2-a93d-6f95a298f19c","Type":"ContainerStarted","Data":"ae12054180de3e8eab4f3f4e006e317d4d1676e2f91b57e2c0f223a868343633"}
Sep 10 19:28:04.463326 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:04.463269    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7" podStartSLOduration=1.463250148 podStartE2EDuration="1.463250148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:04.462491081 +0000 UTC m=+1020.122288274" watchObservedRunningTime="2026-09-10 19:28:04.463250148 +0000 UTC m=+1020.123047344"
Sep 10 19:28:08.389352 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:08.389317    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:28:09.440573 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:09.440542    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:28:15.442566 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:15.442531    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:28:23.308536 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.308497    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"]
Sep 10 19:28:23.312616 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.312588    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.316071 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.316042    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f15c09-serving-cert\""
Sep 10 19:28:23.325553 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.325522    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"]
Sep 10 19:28:23.395809 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.395751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.396009 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.395841    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpfnm\" (UniqueName: \"kubernetes.io/projected/ba88c0a7-40fa-4ded-b076-89fce928063b-kube-api-access-bpfnm\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.396009 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.395869    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.396009 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.395963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba88c0a7-40fa-4ded-b076-89fce928063b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.497185 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.497111    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.497340 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.497197    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfnm\" (UniqueName: \"kubernetes.io/projected/ba88c0a7-40fa-4ded-b076-89fce928063b-kube-api-access-bpfnm\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.497340 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.497241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.497340 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.497281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba88c0a7-40fa-4ded-b076-89fce928063b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.497477 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:28:23.497367    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-serving-cert: secret "maas-api-e2e-shared-b-w3-f15c09-serving-cert" not found
Sep 10 19:28:23.497477 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:28:23.497474    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-maas-api-tls podName:ba88c0a7-40fa-4ded-b076-89fce928063b nodeName:}" failed. No retries permitted until 2026-09-10 19:28:23.997457362 +0000 UTC m=+1039.657254561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5" (UID: "ba88c0a7-40fa-4ded-b076-89fce928063b") : secret "maas-api-e2e-shared-b-w3-f15c09-serving-cert" not found
Sep 10 19:28:23.498035 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.498013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba88c0a7-40fa-4ded-b076-89fce928063b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.499767 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.499742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:23.509010 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:23.508978    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpfnm\" (UniqueName: \"kubernetes.io/projected/ba88c0a7-40fa-4ded-b076-89fce928063b-kube-api-access-bpfnm\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:24.002600 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:24.002559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:24.005335 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:24.005301    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:24.224514 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:24.224472    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:24.507055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:24.506869    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"]
Sep 10 19:28:24.510216 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:28:24.510185    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba88c0a7_40fa_4ded_b076_89fce928063b.slice/crio-bd259ad319c49dbbbb4df4e5099fe57259d96143819c3a4cce9337cb58a9be0a WatchSource:0}: Error finding container bd259ad319c49dbbbb4df4e5099fe57259d96143819c3a4cce9337cb58a9be0a: Status 404 returned error can't find the container with id bd259ad319c49dbbbb4df4e5099fe57259d96143819c3a4cce9337cb58a9be0a
Sep 10 19:28:24.553954 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:24.553862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5" event={"ID":"ba88c0a7-40fa-4ded-b076-89fce928063b","Type":"ContainerStarted","Data":"bd259ad319c49dbbbb4df4e5099fe57259d96143819c3a4cce9337cb58a9be0a"}
Sep 10 19:28:25.560922 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:25.560885    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5" event={"ID":"ba88c0a7-40fa-4ded-b076-89fce928063b","Type":"ContainerStarted","Data":"662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d"}
Sep 10 19:28:25.561381 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:25.561079    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:25.579968 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:25.579912    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5" podStartSLOduration=2.243507964 podStartE2EDuration="2.579893752s" totalImagesPullingTime="336.385788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:25.578088256 +0000 UTC m=+1041.237885450" watchObservedRunningTime="2026-09-10 19:28:25.579893752 +0000 UTC m=+1041.239690946"
Sep 10 19:28:31.572357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:31.572322    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:28:47.016428 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.016394    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"]
Sep 10 19:28:47.020541 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.020517    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.025698 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.025667    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-84cb5814-serving-cert\""
Sep 10 19:28:47.042049 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.041997    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"]
Sep 10 19:28:47.129554 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.129508    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654b27a1-24ae-4e6c-b62d-352832671348-openshift-service-ca\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.129554 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.129561    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmc2w\" (UniqueName: \"kubernetes.io/projected/654b27a1-24ae-4e6c-b62d-352832671348-kube-api-access-rmc2w\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.129803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.129592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-metrics-certs\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.129803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.129719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.231094 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.231048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.231317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.231178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654b27a1-24ae-4e6c-b62d-352832671348-openshift-service-ca\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.231317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.231223    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmc2w\" (UniqueName: \"kubernetes.io/projected/654b27a1-24ae-4e6c-b62d-352832671348-kube-api-access-rmc2w\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.231317 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.231267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-metrics-certs\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.231939 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.231889    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654b27a1-24ae-4e6c-b62d-352832671348-openshift-service-ca\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.234062 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.234042    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.234183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.234113    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-metrics-certs\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.241868 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.241840    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmc2w\" (UniqueName: \"kubernetes.io/projected/654b27a1-24ae-4e6c-b62d-352832671348-kube-api-access-rmc2w\") pod \"maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.331279 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.331242    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:47.610433 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.610402    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"]
Sep 10 19:28:47.612200 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:28:47.612172    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654b27a1_24ae_4e6c_b62d_352832671348.slice/crio-64e4ec0019d07bffc6225a79d58b328aa883ae7c75d0350eebd7332ed35c9e4c WatchSource:0}: Error finding container 64e4ec0019d07bffc6225a79d58b328aa883ae7c75d0350eebd7332ed35c9e4c: Status 404 returned error can't find the container with id 64e4ec0019d07bffc6225a79d58b328aa883ae7c75d0350eebd7332ed35c9e4c
Sep 10 19:28:47.693240 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:47.693199    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf" event={"ID":"654b27a1-24ae-4e6c-b62d-352832671348","Type":"ContainerStarted","Data":"64e4ec0019d07bffc6225a79d58b328aa883ae7c75d0350eebd7332ed35c9e4c"}
Sep 10 19:28:48.700848 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:48.700808    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf" event={"ID":"654b27a1-24ae-4e6c-b62d-352832671348","Type":"ContainerStarted","Data":"18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6"}
Sep 10 19:28:48.701273 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:48.700939    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:28:48.752573 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:48.752515    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf" podStartSLOduration=2.305585813 podStartE2EDuration="2.752494992s" totalImagesPullingTime="446.909179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:48.750358031 +0000 UTC m=+1064.410155225" watchObservedRunningTime="2026-09-10 19:28:48.752494992 +0000 UTC m=+1064.412292188"
Sep 10 19:28:54.711574 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:28:54.711538    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:30:05.832274 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:05.832238    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"]
Sep 10 19:30:05.936419 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:30:05.936385    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-serving-cert: secret "maas-api-e2e-mt-84cb5814-serving-cert" not found
Sep 10 19:30:05.936736 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:30:05.936710    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls podName:654b27a1-24ae-4e6c-b62d-352832671348 nodeName:}" failed. No retries permitted until 2026-09-10 19:30:06.436681115 +0000 UTC m=+1142.096478302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls") pod "maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf" (UID: "654b27a1-24ae-4e6c-b62d-352832671348") : secret "maas-api-e2e-mt-84cb5814-serving-cert" not found
Sep 10 19:30:06.133270 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.133163    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf" podUID="654b27a1-24ae-4e6c-b62d-352832671348" containerName="maas-api" containerID="cri-o://18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6" gracePeriod=30
Sep 10 19:30:06.391522 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.391444    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:30:06.441806 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:30:06.441768    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-serving-cert: secret "maas-api-e2e-mt-84cb5814-serving-cert" not found
Sep 10 19:30:06.441987 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:30:06.441844    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls podName:654b27a1-24ae-4e6c-b62d-352832671348 nodeName:}" failed. No retries permitted until 2026-09-10 19:30:07.441828376 +0000 UTC m=+1143.101625547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls") pod "maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf" (UID: "654b27a1-24ae-4e6c-b62d-352832671348") : secret "maas-api-e2e-mt-84cb5814-serving-cert" not found
Sep 10 19:30:06.542534 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.542490    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654b27a1-24ae-4e6c-b62d-352832671348-openshift-service-ca\") pod \"654b27a1-24ae-4e6c-b62d-352832671348\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") "
Sep 10 19:30:06.542714 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.542552    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls\") pod \"654b27a1-24ae-4e6c-b62d-352832671348\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") "
Sep 10 19:30:06.542714 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.542662    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmc2w\" (UniqueName: \"kubernetes.io/projected/654b27a1-24ae-4e6c-b62d-352832671348-kube-api-access-rmc2w\") pod \"654b27a1-24ae-4e6c-b62d-352832671348\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") "
Sep 10 19:30:06.542714 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.542678    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-metrics-certs\") pod \"654b27a1-24ae-4e6c-b62d-352832671348\" (UID: \"654b27a1-24ae-4e6c-b62d-352832671348\") "
Sep 10 19:30:06.542969 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.542941    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654b27a1-24ae-4e6c-b62d-352832671348-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "654b27a1-24ae-4e6c-b62d-352832671348" (UID: "654b27a1-24ae-4e6c-b62d-352832671348"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:30:06.545043 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.545005    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "654b27a1-24ae-4e6c-b62d-352832671348" (UID: "654b27a1-24ae-4e6c-b62d-352832671348"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:30:06.545225 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.545015    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "654b27a1-24ae-4e6c-b62d-352832671348" (UID: "654b27a1-24ae-4e6c-b62d-352832671348"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:30:06.545225 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.545048    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654b27a1-24ae-4e6c-b62d-352832671348-kube-api-access-rmc2w" (OuterVolumeSpecName: "kube-api-access-rmc2w") pod "654b27a1-24ae-4e6c-b62d-352832671348" (UID: "654b27a1-24ae-4e6c-b62d-352832671348"). InnerVolumeSpecName "kube-api-access-rmc2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:30:06.644467 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.644361    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmc2w\" (UniqueName: \"kubernetes.io/projected/654b27a1-24ae-4e6c-b62d-352832671348-kube-api-access-rmc2w\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:30:06.644467 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.644396    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:30:06.644467 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.644411    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/654b27a1-24ae-4e6c-b62d-352832671348-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:30:06.644467 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:06.644422    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/654b27a1-24ae-4e6c-b62d-352832671348-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:30:07.139854 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:07.139817    2573 generic.go:356] "Generic (PLEG): container finished" podID="654b27a1-24ae-4e6c-b62d-352832671348" containerID="18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6" exitCode=0
Sep 10 19:30:07.140343 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:07.139882    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"
Sep 10 19:30:07.140343 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:07.139907    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf" event={"ID":"654b27a1-24ae-4e6c-b62d-352832671348","Type":"ContainerDied","Data":"18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6"}
Sep 10 19:30:07.140343 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:07.139972    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf" event={"ID":"654b27a1-24ae-4e6c-b62d-352832671348","Type":"ContainerDied","Data":"64e4ec0019d07bffc6225a79d58b328aa883ae7c75d0350eebd7332ed35c9e4c"}
Sep 10 19:30:07.140343 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:07.139997    2573 scope.go:117] "RemoveContainer" containerID="18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6"
Sep 10 19:30:07.152577 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:07.152554    2573 scope.go:117] "RemoveContainer" containerID="18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6"
Sep 10 19:30:07.152929 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:30:07.152903    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6\": container with ID starting with 18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6 not found: ID does not exist" containerID="18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6"
Sep 10 19:30:07.153018 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:07.152938    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6"} err="failed to get container status \"18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6\": rpc error: code = NotFound desc = could not find container \"18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6\": container with ID starting with 18a627a155d0e9ea402a13d535102ab8fce6aae3d883258efe87aad901b98fa6 not found: ID does not exist"
Sep 10 19:30:07.159655 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:07.159623    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"]
Sep 10 19:30:07.164404 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:07.164375    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84cb5814-5458bc8544-9rnzf"]
Sep 10 19:30:08.835234 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:30:08.835196    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654b27a1-24ae-4e6c-b62d-352832671348" path="/var/lib/kubelet/pods/654b27a1-24ae-4e6c-b62d-352832671348/volumes"
Sep 10 19:32:02.624726 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.624692    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"]
Sep 10 19:32:02.625245 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.625219    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654b27a1-24ae-4e6c-b62d-352832671348" containerName="maas-api"
Sep 10 19:32:02.625245 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.625243    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="654b27a1-24ae-4e6c-b62d-352832671348" containerName="maas-api"
Sep 10 19:32:02.625464 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.625376    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="654b27a1-24ae-4e6c-b62d-352832671348" containerName="maas-api"
Sep 10 19:32:02.628478 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.628461    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.633312 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.633286    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"default-dockercfg-5k7v8\""
Sep 10 19:32:02.633439 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.633287    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"sub-test-model-b64b1c-kserve-self-signed-certs\""
Sep 10 19:32:02.633439 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.633333    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"kube-root-ca.crt\""
Sep 10 19:32:02.633439 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.633312    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"openshift-service-ca.crt\""
Sep 10 19:32:02.648092 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.648058    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"]
Sep 10 19:32:02.785790 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.785746    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-model-cache\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.785790 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.785796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-tmp-dir\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.786020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.785819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6zjq\" (UniqueName: \"kubernetes.io/projected/b2ef0f87-d75e-4748-af10-f9d661aad945-kube-api-access-g6zjq\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.786020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.785846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-home\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.786020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.785947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-dshm\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.786020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.785981    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ef0f87-d75e-4748-af10-f9d661aad945-tls-certs\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.887422 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.887319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-dshm\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.887422 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.887381    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ef0f87-d75e-4748-af10-f9d661aad945-tls-certs\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.887649 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.887430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-model-cache\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.887649 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.887450    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-tmp-dir\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.887649 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.887583    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6zjq\" (UniqueName: \"kubernetes.io/projected/b2ef0f87-d75e-4748-af10-f9d661aad945-kube-api-access-g6zjq\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.887649 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.887618    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-home\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.887885 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.887848    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-tmp-dir\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.887969 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.887895    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-model-cache\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.888032 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.887993    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-home\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.889894 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.889857    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-dshm\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.890127 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.890107    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ef0f87-d75e-4748-af10-f9d661aad945-tls-certs\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.896597 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.896570    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6zjq\" (UniqueName: \"kubernetes.io/projected/b2ef0f87-d75e-4748-af10-f9d661aad945-kube-api-access-g6zjq\") pod \"sub-test-model-b64b1c-kserve-864b9dcddc-99ssx\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:02.951329 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:02.951288    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:03.228352 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:03.228329    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"]
Sep 10 19:32:03.230809 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:32:03.230782    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2ef0f87_d75e_4748_af10_f9d661aad945.slice/crio-68a58bdef548eadfab9dc5c03116a6a76e6233d1bd3c72cfe849fc67becb9459 WatchSource:0}: Error finding container 68a58bdef548eadfab9dc5c03116a6a76e6233d1bd3c72cfe849fc67becb9459: Status 404 returned error can't find the container with id 68a58bdef548eadfab9dc5c03116a6a76e6233d1bd3c72cfe849fc67becb9459
Sep 10 19:32:03.775603 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:03.775564    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx" event={"ID":"b2ef0f87-d75e-4748-af10-f9d661aad945","Type":"ContainerStarted","Data":"a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5"}
Sep 10 19:32:03.775997 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:03.775617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx" event={"ID":"b2ef0f87-d75e-4748-af10-f9d661aad945","Type":"ContainerStarted","Data":"68a58bdef548eadfab9dc5c03116a6a76e6233d1bd3c72cfe849fc67becb9459"}
Sep 10 19:32:03.796528 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:03.796461    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx" podStartSLOduration=1.796439907 podStartE2EDuration="1.796439907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:03.793911396 +0000 UTC m=+1259.453708592" watchObservedRunningTime="2026-09-10 19:32:03.796439907 +0000 UTC m=+1259.456237100"
Sep 10 19:32:12.952135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:12.952095    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:12.954634 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:12.952182    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:12.964859 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:12.964832    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:13.839938 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:13.839910    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:32:20.692494 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.692463    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"]
Sep 10 19:32:20.696261 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.696235    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.699130 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.699100    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cd7dca-serving-cert\""
Sep 10 19:32:20.708544 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.708517    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"]
Sep 10 19:32:20.763433 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.763389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-metrics-certs\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.763633 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.763464    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.763633 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.763544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5xr6\" (UniqueName: \"kubernetes.io/projected/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-kube-api-access-w5xr6\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.763633 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.763579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-maas-api-tls\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.864327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.864292    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-metrics-certs\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.864568 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.864372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.864568 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.864444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5xr6\" (UniqueName: \"kubernetes.io/projected/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-kube-api-access-w5xr6\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.864568 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.864476    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-maas-api-tls\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.865305 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.865283    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-openshift-service-ca\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.867257 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.867235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-metrics-certs\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.867355 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.867277    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-maas-api-tls\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:20.886118 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:20.886084    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5xr6\" (UniqueName: \"kubernetes.io/projected/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-kube-api-access-w5xr6\") pod \"maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:21.007581 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:21.007496    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:21.298635 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:21.298605    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"]
Sep 10 19:32:21.300481 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:32:21.300435    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0b307f_ebca_47a3_9fd3_3a31dbd45ce1.slice/crio-07ad0cd2c0d8ada4bf566d8afc7dd45101a499669f8ddb250b31fe0f7b0c8a45 WatchSource:0}: Error finding container 07ad0cd2c0d8ada4bf566d8afc7dd45101a499669f8ddb250b31fe0f7b0c8a45: Status 404 returned error can't find the container with id 07ad0cd2c0d8ada4bf566d8afc7dd45101a499669f8ddb250b31fe0f7b0c8a45
Sep 10 19:32:21.873654 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:21.873621    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp" event={"ID":"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1","Type":"ContainerStarted","Data":"dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589"}
Sep 10 19:32:21.874046 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:21.873679    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp" event={"ID":"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1","Type":"ContainerStarted","Data":"07ad0cd2c0d8ada4bf566d8afc7dd45101a499669f8ddb250b31fe0f7b0c8a45"}
Sep 10 19:32:21.874046 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:21.873723    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:21.894397 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:21.894345    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp" podStartSLOduration=1.50157682 podStartE2EDuration="1.89432986s" totalImagesPullingTime="392.75304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:21.893218536 +0000 UTC m=+1277.553015729" watchObservedRunningTime="2026-09-10 19:32:21.89432986 +0000 UTC m=+1277.554127054"
Sep 10 19:32:27.885419 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:27.885387    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:32:49.935372 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:49.935331    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"]
Sep 10 19:32:49.939468 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:49.939446    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:32:49.942104 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:49.942078    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-66e9b4-dockercfg-fdq7d\""
Sep 10 19:32:49.942240 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:49.942083    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-66e9b4\""
Sep 10 19:32:49.952333 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:49.952310    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"]
Sep 10 19:32:49.959860 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:49.959825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm\" (UID: \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:32:49.960017 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:49.959889    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fp6\" (UniqueName: \"kubernetes.io/projected/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-kube-api-access-q7fp6\") pod \"payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm\" (UID: \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:32:50.061348 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:50.061317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fp6\" (UniqueName: \"kubernetes.io/projected/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-kube-api-access-q7fp6\") pod \"payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm\" (UID: \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:32:50.061552 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:50.061419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm\" (UID: \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:32:50.062042 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:50.062021    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm\" (UID: \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:32:50.070725 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:50.070696    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fp6\" (UniqueName: \"kubernetes.io/projected/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-kube-api-access-q7fp6\") pod \"payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm\" (UID: \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:32:50.251377 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:50.251278    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:32:50.535527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:50.535494    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"]
Sep 10 19:32:50.536922 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:32:50.536891    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff493a16_2e8a_4ac0_969b_f492eca8c5d4.slice/crio-84e0ae7c587ec5cf6ab0ff5080f0c2e33297215c56908141817caeb77ad108a5 WatchSource:0}: Error finding container 84e0ae7c587ec5cf6ab0ff5080f0c2e33297215c56908141817caeb77ad108a5: Status 404 returned error can't find the container with id 84e0ae7c587ec5cf6ab0ff5080f0c2e33297215c56908141817caeb77ad108a5
Sep 10 19:32:51.031865 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:51.031819    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm" event={"ID":"ff493a16-2e8a-4ac0-969b-f492eca8c5d4","Type":"ContainerStarted","Data":"26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a"}
Sep 10 19:32:51.032281 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:51.031881    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm" event={"ID":"ff493a16-2e8a-4ac0-969b-f492eca8c5d4","Type":"ContainerStarted","Data":"84e0ae7c587ec5cf6ab0ff5080f0c2e33297215c56908141817caeb77ad108a5"}
Sep 10 19:32:51.032281 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:51.031965    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:32:51.050889 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:32:51.050836    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm" podStartSLOduration=2.05081973 podStartE2EDuration="2.05081973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:51.048836752 +0000 UTC m=+1306.708633946" watchObservedRunningTime="2026-09-10 19:32:51.05081973 +0000 UTC m=+1306.710616927"
Sep 10 19:33:02.041034 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:02.041000    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:33:31.434867 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.434834    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"]
Sep 10 19:33:31.435422 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.435043    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp" podUID="1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" containerName="maas-api" containerID="cri-o://dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589" gracePeriod=30
Sep 10 19:33:31.687535 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.687469    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:33:31.789233 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.789197    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-maas-api-tls\") pod \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") "
Sep 10 19:33:31.789408 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.789274    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-metrics-certs\") pod \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") "
Sep 10 19:33:31.789408 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.789301    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5xr6\" (UniqueName: \"kubernetes.io/projected/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-kube-api-access-w5xr6\") pod \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") "
Sep 10 19:33:31.789489 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.789407    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-openshift-service-ca\") pod \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\" (UID: \"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1\") "
Sep 10 19:33:31.789879 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.789842    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" (UID: "1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:33:31.791339 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.791312    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" (UID: "1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:33:31.791552 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.791530    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-kube-api-access-w5xr6" (OuterVolumeSpecName: "kube-api-access-w5xr6") pod "1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" (UID: "1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1"). InnerVolumeSpecName "kube-api-access-w5xr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:33:31.791603 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.791536    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" (UID: "1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:33:31.891019 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.890985    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:31.891019 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.891014    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5xr6\" (UniqueName: \"kubernetes.io/projected/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-kube-api-access-w5xr6\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:31.891019 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.891023    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:31.891295 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:31.891032    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:32.275913 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.275884    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" containerID="dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589" exitCode=0
Sep 10 19:33:32.276138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.275933    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"
Sep 10 19:33:32.276138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.275965    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp" event={"ID":"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1","Type":"ContainerDied","Data":"dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589"}
Sep 10 19:33:32.276138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.276014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp" event={"ID":"1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1","Type":"ContainerDied","Data":"07ad0cd2c0d8ada4bf566d8afc7dd45101a499669f8ddb250b31fe0f7b0c8a45"}
Sep 10 19:33:32.276138 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.276034    2573 scope.go:117] "RemoveContainer" containerID="dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589"
Sep 10 19:33:32.286893 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.286877    2573 scope.go:117] "RemoveContainer" containerID="dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589"
Sep 10 19:33:32.287167 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:33:32.287130    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589\": container with ID starting with dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589 not found: ID does not exist" containerID="dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589"
Sep 10 19:33:32.287221 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.287176    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589"} err="failed to get container status \"dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589\": rpc error: code = NotFound desc = could not find container \"dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589\": container with ID starting with dc4023de1559426cef9fe2b0b808d9acc399f203d0f0a301f0b134c8a0abb589 not found: ID does not exist"
Sep 10 19:33:32.300751 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.300721    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"]
Sep 10 19:33:32.303714 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.303692    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cd7dca-dfbb5c94-m76sp"]
Sep 10 19:33:32.834349 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:32.834311    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" path="/var/lib/kubelet/pods/1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1/volumes"
Sep 10 19:33:34.554840 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.554801    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7528-f52733"]
Sep 10 19:33:34.555336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.555321    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" containerName="maas-api"
Sep 10 19:33:34.555393 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.555338    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" containerName="maas-api"
Sep 10 19:33:34.555436 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.555429    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b0b307f-ebca-47a3-9fd3-3a31dbd45ce1" containerName="maas-api"
Sep 10 19:33:34.558540 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.558524    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-f52733"
Sep 10 19:33:34.561280 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.561257    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sbhkd\""
Sep 10 19:33:34.566077 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.566051    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-f52733"]
Sep 10 19:33:34.619489 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.619446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbr4c\" (UniqueName: \"kubernetes.io/projected/da2d030a-7631-48ce-8df7-64fb24e2c008-kube-api-access-rbr4c\") pod \"test-curl-7528-f52733\" (UID: \"da2d030a-7631-48ce-8df7-64fb24e2c008\") " pod="opendatahub/test-curl-7528-f52733"
Sep 10 19:33:34.720917 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.720876    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbr4c\" (UniqueName: \"kubernetes.io/projected/da2d030a-7631-48ce-8df7-64fb24e2c008-kube-api-access-rbr4c\") pod \"test-curl-7528-f52733\" (UID: \"da2d030a-7631-48ce-8df7-64fb24e2c008\") " pod="opendatahub/test-curl-7528-f52733"
Sep 10 19:33:34.729675 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.729650    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbr4c\" (UniqueName: \"kubernetes.io/projected/da2d030a-7631-48ce-8df7-64fb24e2c008-kube-api-access-rbr4c\") pod \"test-curl-7528-f52733\" (UID: \"da2d030a-7631-48ce-8df7-64fb24e2c008\") " pod="opendatahub/test-curl-7528-f52733"
Sep 10 19:33:34.869338 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:34.869252    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-f52733"
Sep 10 19:33:35.144036 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:35.144010    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-f52733"]
Sep 10 19:33:35.146202 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:33:35.146171    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda2d030a_7631_48ce_8df7_64fb24e2c008.slice/crio-6c7f5786869b16e0fe3a3cd0fb5cc7441fc31819ed1a988537a44e91d00d2029 WatchSource:0}: Error finding container 6c7f5786869b16e0fe3a3cd0fb5cc7441fc31819ed1a988537a44e91d00d2029: Status 404 returned error can't find the container with id 6c7f5786869b16e0fe3a3cd0fb5cc7441fc31819ed1a988537a44e91d00d2029
Sep 10 19:33:35.295232 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:35.295191    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-f52733" event={"ID":"da2d030a-7631-48ce-8df7-64fb24e2c008","Type":"ContainerStarted","Data":"a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b"}
Sep 10 19:33:35.295420 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:35.295246    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-f52733" event={"ID":"da2d030a-7631-48ce-8df7-64fb24e2c008","Type":"ContainerStarted","Data":"6c7f5786869b16e0fe3a3cd0fb5cc7441fc31819ed1a988537a44e91d00d2029"}
Sep 10 19:33:35.312518 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:35.312468    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7528-f52733" podStartSLOduration=1.31245179 podStartE2EDuration="1.31245179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:33:35.31210848 +0000 UTC m=+1350.971905673" watchObservedRunningTime="2026-09-10 19:33:35.31245179 +0000 UTC m=+1350.972248987"
Sep 10 19:33:35.931356 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:35.931325    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7528-f52733"]
Sep 10 19:33:35.937453 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:35.937424    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7528-f52733"]
Sep 10 19:33:36.301061 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:33:36.301029    2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7528-f52733" container="test-curl-7528-f52733"
Sep 10 19:33:37.314349 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:37.314288    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7528-f52733" podUID="da2d030a-7631-48ce-8df7-64fb24e2c008" containerName="test-curl-7528-f52733" containerID="cri-o://a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b" gracePeriod=2
Sep 10 19:33:39.491812 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:39.491780    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-f52733"
Sep 10 19:33:39.564902 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:39.564856    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbr4c\" (UniqueName: \"kubernetes.io/projected/da2d030a-7631-48ce-8df7-64fb24e2c008-kube-api-access-rbr4c\") pod \"da2d030a-7631-48ce-8df7-64fb24e2c008\" (UID: \"da2d030a-7631-48ce-8df7-64fb24e2c008\") "
Sep 10 19:33:39.567304 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:39.567271    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2d030a-7631-48ce-8df7-64fb24e2c008-kube-api-access-rbr4c" (OuterVolumeSpecName: "kube-api-access-rbr4c") pod "da2d030a-7631-48ce-8df7-64fb24e2c008" (UID: "da2d030a-7631-48ce-8df7-64fb24e2c008"). InnerVolumeSpecName "kube-api-access-rbr4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:33:39.666336 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:39.666303    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbr4c\" (UniqueName: \"kubernetes.io/projected/da2d030a-7631-48ce-8df7-64fb24e2c008-kube-api-access-rbr4c\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:39.680060 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:39.680027    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"]
Sep 10 19:33:40.332953 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:40.332920    2573 generic.go:356] "Generic (PLEG): container finished" podID="da2d030a-7631-48ce-8df7-64fb24e2c008" containerID="a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b" exitCode=137
Sep 10 19:33:40.333142 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:40.332960    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-f52733"
Sep 10 19:33:40.333142 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:40.333018    2573 scope.go:117] "RemoveContainer" containerID="a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b"
Sep 10 19:33:40.342434 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:40.342414    2573 scope.go:117] "RemoveContainer" containerID="a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b"
Sep 10 19:33:40.342702 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:33:40.342686    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b\": container with ID starting with a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b not found: ID does not exist" containerID="a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b"
Sep 10 19:33:40.342753 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:40.342711    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b"} err="failed to get container status \"a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b\": rpc error: code = NotFound desc = could not find container \"a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b\": container with ID starting with a1d1a798c2b96ac446dd7e0a2d50c8ab4d8d894c79f9eddd0f91228767fa5d9b not found: ID does not exist"
Sep 10 19:33:40.343400 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:33:40.343378    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7528-f52733\" is forbidden: User \"system:node:ip-10-0-141-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-45.ec2.internal' and this object" podUID="da2d030a-7631-48ce-8df7-64fb24e2c008" pod="opendatahub/test-curl-7528-f52733"
Sep 10 19:33:40.833840 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:40.833809    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da2d030a-7631-48ce-8df7-64fb24e2c008" path="/var/lib/kubelet/pods/da2d030a-7631-48ce-8df7-64fb24e2c008/volumes"
Sep 10 19:33:54.701047 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:54.700989    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx" podUID="b2ef0f87-d75e-4748-af10-f9d661aad945" containerName="main" containerID="cri-o://a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5" gracePeriod=45
Sep 10 19:33:54.915421 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:54.915388    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"]
Sep 10 19:33:54.915620 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:54.915595    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm" podUID="ff493a16-2e8a-4ac0-969b-f492eca8c5d4" containerName="payload-pre-processing" containerID="cri-o://26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a" gracePeriod=30
Sep 10 19:33:55.048425 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.048386    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:33:55.174453 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.174427    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:33:55.223626 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.223594    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-model-cache\") pod \"b2ef0f87-d75e-4748-af10-f9d661aad945\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") "
Sep 10 19:33:55.223626 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.223632    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-home\") pod \"b2ef0f87-d75e-4748-af10-f9d661aad945\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") "
Sep 10 19:33:55.223893 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.223657    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ef0f87-d75e-4748-af10-f9d661aad945-tls-certs\") pod \"b2ef0f87-d75e-4748-af10-f9d661aad945\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") "
Sep 10 19:33:55.223893 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.223770    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6zjq\" (UniqueName: \"kubernetes.io/projected/b2ef0f87-d75e-4748-af10-f9d661aad945-kube-api-access-g6zjq\") pod \"b2ef0f87-d75e-4748-af10-f9d661aad945\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") "
Sep 10 19:33:55.223893 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.223799    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-tmp-dir\") pod \"b2ef0f87-d75e-4748-af10-f9d661aad945\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") "
Sep 10 19:33:55.223893 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.223852    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-dshm\") pod \"b2ef0f87-d75e-4748-af10-f9d661aad945\" (UID: \"b2ef0f87-d75e-4748-af10-f9d661aad945\") "
Sep 10 19:33:55.223893 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.223870    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-home" (OuterVolumeSpecName: "home") pod "b2ef0f87-d75e-4748-af10-f9d661aad945" (UID: "b2ef0f87-d75e-4748-af10-f9d661aad945"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:55.224273 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.223928    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2ef0f87-d75e-4748-af10-f9d661aad945" (UID: "b2ef0f87-d75e-4748-af10-f9d661aad945"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:55.224273 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.224122    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2ef0f87-d75e-4748-af10-f9d661aad945" (UID: "b2ef0f87-d75e-4748-af10-f9d661aad945"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:55.224273 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.224202    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.224273 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.224218    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.226269 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.226241    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ef0f87-d75e-4748-af10-f9d661aad945-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2ef0f87-d75e-4748-af10-f9d661aad945" (UID: "b2ef0f87-d75e-4748-af10-f9d661aad945"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:33:55.226269 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.226241    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ef0f87-d75e-4748-af10-f9d661aad945-kube-api-access-g6zjq" (OuterVolumeSpecName: "kube-api-access-g6zjq") pod "b2ef0f87-d75e-4748-af10-f9d661aad945" (UID: "b2ef0f87-d75e-4748-af10-f9d661aad945"). InnerVolumeSpecName "kube-api-access-g6zjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:33:55.226443 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.226428    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-dshm" (OuterVolumeSpecName: "dshm") pod "b2ef0f87-d75e-4748-af10-f9d661aad945" (UID: "b2ef0f87-d75e-4748-af10-f9d661aad945"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:55.325201 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.325139    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-plugins-config-volume\") pod \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\" (UID: \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\") "
Sep 10 19:33:55.325383 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.325305    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7fp6\" (UniqueName: \"kubernetes.io/projected/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-kube-api-access-q7fp6\") pod \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\" (UID: \"ff493a16-2e8a-4ac0-969b-f492eca8c5d4\") "
Sep 10 19:33:55.325468 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.325424    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff493a16-2e8a-4ac0-969b-f492eca8c5d4" (UID: "ff493a16-2e8a-4ac0-969b-f492eca8c5d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:33:55.325629 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.325609    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6zjq\" (UniqueName: \"kubernetes.io/projected/b2ef0f87-d75e-4748-af10-f9d661aad945-kube-api-access-g6zjq\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.325689 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.325637    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.325689 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.325651    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2ef0f87-d75e-4748-af10-f9d661aad945-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.325689 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.325666    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2ef0f87-d75e-4748-af10-f9d661aad945-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.325689 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.325680    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.327491 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.327469    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-kube-api-access-q7fp6" (OuterVolumeSpecName: "kube-api-access-q7fp6") pod "ff493a16-2e8a-4ac0-969b-f492eca8c5d4" (UID: "ff493a16-2e8a-4ac0-969b-f492eca8c5d4"). InnerVolumeSpecName "kube-api-access-q7fp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:33:55.417969 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.417935    2573 generic.go:356] "Generic (PLEG): container finished" podID="b2ef0f87-d75e-4748-af10-f9d661aad945" containerID="a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5" exitCode=0
Sep 10 19:33:55.418183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.417985    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"
Sep 10 19:33:55.418183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.418025    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx" event={"ID":"b2ef0f87-d75e-4748-af10-f9d661aad945","Type":"ContainerDied","Data":"a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5"}
Sep 10 19:33:55.418183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.418075    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx" event={"ID":"b2ef0f87-d75e-4748-af10-f9d661aad945","Type":"ContainerDied","Data":"68a58bdef548eadfab9dc5c03116a6a76e6233d1bd3c72cfe849fc67becb9459"}
Sep 10 19:33:55.418183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.418095    2573 scope.go:117] "RemoveContainer" containerID="a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5"
Sep 10 19:33:55.419574 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.419548    2573 generic.go:356] "Generic (PLEG): container finished" podID="ff493a16-2e8a-4ac0-969b-f492eca8c5d4" containerID="26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a" exitCode=0
Sep 10 19:33:55.419697 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.419595    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"
Sep 10 19:33:55.419697 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.419595    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm" event={"ID":"ff493a16-2e8a-4ac0-969b-f492eca8c5d4","Type":"ContainerDied","Data":"26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a"}
Sep 10 19:33:55.419991 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.419743    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm" event={"ID":"ff493a16-2e8a-4ac0-969b-f492eca8c5d4","Type":"ContainerDied","Data":"84e0ae7c587ec5cf6ab0ff5080f0c2e33297215c56908141817caeb77ad108a5"}
Sep 10 19:33:55.426143 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.426121    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7fp6\" (UniqueName: \"kubernetes.io/projected/ff493a16-2e8a-4ac0-969b-f492eca8c5d4-kube-api-access-q7fp6\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.430743 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.430725    2573 scope.go:117] "RemoveContainer" containerID="a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5"
Sep 10 19:33:55.431258 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:33:55.431234    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5\": container with ID starting with a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5 not found: ID does not exist" containerID="a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5"
Sep 10 19:33:55.431354 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.431277    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5"} err="failed to get container status \"a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5\": rpc error: code = NotFound desc = could not find container \"a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5\": container with ID starting with a95b81201a97cde9a4df458d2e9011b7aa83d2fdf624a5f0b69e9597c179b1e5 not found: ID does not exist"
Sep 10 19:33:55.431354 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.431315    2573 scope.go:117] "RemoveContainer" containerID="26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a"
Sep 10 19:33:55.440724 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.440703    2573 scope.go:117] "RemoveContainer" containerID="26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a"
Sep 10 19:33:55.441032 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:33:55.441009    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a\": container with ID starting with 26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a not found: ID does not exist" containerID="26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a"
Sep 10 19:33:55.441077 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.441040    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a"} err="failed to get container status \"26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a\": rpc error: code = NotFound desc = could not find container \"26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a\": container with ID starting with 26843dac2a4c3f577dfc61383db29cd6eac53737f513956cb5463c4f2694db7a not found: ID does not exist"
Sep 10 19:33:55.457891 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.457863    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"]
Sep 10 19:33:55.467631 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.467591    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-66e9b4-67cf87f654-5bznm"]
Sep 10 19:33:55.523390 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.523364    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"]
Sep 10 19:33:55.530393 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:55.530367    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/sub-test-model-b64b1c-kserve-864b9dcddc-99ssx"]
Sep 10 19:33:56.834669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:56.834634    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ef0f87-d75e-4748-af10-f9d661aad945" path="/var/lib/kubelet/pods/b2ef0f87-d75e-4748-af10-f9d661aad945/volumes"
Sep 10 19:33:56.835054 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:33:56.834994    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff493a16-2e8a-4ac0-969b-f492eca8c5d4" path="/var/lib/kubelet/pods/ff493a16-2e8a-4ac0-969b-f492eca8c5d4/volumes"
Sep 10 19:34:05.037126 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:05.037095    2573 scope.go:117] "RemoveContainer" containerID="224de71471ebc7eb7e52a784d92c89455d7798a6df16a705e1fcab4f01123469"
Sep 10 19:34:05.045892 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:05.045871    2573 scope.go:117] "RemoveContainer" containerID="b90b1edc8d5cf1a30e49ec44a442d2639f617fe45dc791b1fd25dcd9c2274ade"
Sep 10 19:34:13.643480 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:13.643441    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"]
Sep 10 19:34:13.643908 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:13.643693    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd" podUID="63d35b8f-38f0-447b-b60e-2108d67ceb40" containerName="maas-api" containerID="cri-o://26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188" gracePeriod=30
Sep 10 19:34:13.708787 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:13.708749    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"]
Sep 10 19:34:13.709128 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:13.709074    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q" podUID="0156c21b-afca-4717-920f-62ffff92a259" containerName="payload-processing" containerID="cri-o://ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d" gracePeriod=30
Sep 10 19:34:13.959731 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:13.959698    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:34:13.988529 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:13.988498    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:34:14.000454 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.000419    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-metrics-certs\") pod \"63d35b8f-38f0-447b-b60e-2108d67ceb40\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") "
Sep 10 19:34:14.000617 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.000497    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d35b8f-38f0-447b-b60e-2108d67ceb40-openshift-service-ca\") pod \"63d35b8f-38f0-447b-b60e-2108d67ceb40\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") "
Sep 10 19:34:14.000617 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.000541    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-maas-api-tls\") pod \"63d35b8f-38f0-447b-b60e-2108d67ceb40\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") "
Sep 10 19:34:14.000617 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.000577    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbv2n\" (UniqueName: \"kubernetes.io/projected/63d35b8f-38f0-447b-b60e-2108d67ceb40-kube-api-access-vbv2n\") pod \"63d35b8f-38f0-447b-b60e-2108d67ceb40\" (UID: \"63d35b8f-38f0-447b-b60e-2108d67ceb40\") "
Sep 10 19:34:14.000982 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.000893    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d35b8f-38f0-447b-b60e-2108d67ceb40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63d35b8f-38f0-447b-b60e-2108d67ceb40" (UID: "63d35b8f-38f0-447b-b60e-2108d67ceb40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:34:14.002831 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.002804    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d35b8f-38f0-447b-b60e-2108d67ceb40-kube-api-access-vbv2n" (OuterVolumeSpecName: "kube-api-access-vbv2n") pod "63d35b8f-38f0-447b-b60e-2108d67ceb40" (UID: "63d35b8f-38f0-447b-b60e-2108d67ceb40"). InnerVolumeSpecName "kube-api-access-vbv2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:34:14.003270 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.003249    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63d35b8f-38f0-447b-b60e-2108d67ceb40" (UID: "63d35b8f-38f0-447b-b60e-2108d67ceb40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:34:14.003358 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.003271    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "63d35b8f-38f0-447b-b60e-2108d67ceb40" (UID: "63d35b8f-38f0-447b-b60e-2108d67ceb40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:34:14.101552 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.101514    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0156c21b-afca-4717-920f-62ffff92a259-plugins-config-volume\") pod \"0156c21b-afca-4717-920f-62ffff92a259\" (UID: \"0156c21b-afca-4717-920f-62ffff92a259\") "
Sep 10 19:34:14.101772 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.101579    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz22n\" (UniqueName: \"kubernetes.io/projected/0156c21b-afca-4717-920f-62ffff92a259-kube-api-access-sz22n\") pod \"0156c21b-afca-4717-920f-62ffff92a259\" (UID: \"0156c21b-afca-4717-920f-62ffff92a259\") "
Sep 10 19:34:14.101856 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.101831    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:14.101915 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.101861    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63d35b8f-38f0-447b-b60e-2108d67ceb40-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:14.101915 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.101871    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63d35b8f-38f0-447b-b60e-2108d67ceb40-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:14.101915 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.101879    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbv2n\" (UniqueName: \"kubernetes.io/projected/63d35b8f-38f0-447b-b60e-2108d67ceb40-kube-api-access-vbv2n\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:14.101915 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.101898    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0156c21b-afca-4717-920f-62ffff92a259-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0156c21b-afca-4717-920f-62ffff92a259" (UID: "0156c21b-afca-4717-920f-62ffff92a259"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:34:14.103862 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.103836    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0156c21b-afca-4717-920f-62ffff92a259-kube-api-access-sz22n" (OuterVolumeSpecName: "kube-api-access-sz22n") pod "0156c21b-afca-4717-920f-62ffff92a259" (UID: "0156c21b-afca-4717-920f-62ffff92a259"). InnerVolumeSpecName "kube-api-access-sz22n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:34:14.202793 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.202695    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0156c21b-afca-4717-920f-62ffff92a259-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:14.202793 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.202731    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz22n\" (UniqueName: \"kubernetes.io/projected/0156c21b-afca-4717-920f-62ffff92a259-kube-api-access-sz22n\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:14.525727 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.525687    2573 generic.go:356] "Generic (PLEG): container finished" podID="63d35b8f-38f0-447b-b60e-2108d67ceb40" containerID="26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188" exitCode=0
Sep 10 19:34:14.525898 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.525734    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"
Sep 10 19:34:14.525898 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.525764    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd" event={"ID":"63d35b8f-38f0-447b-b60e-2108d67ceb40","Type":"ContainerDied","Data":"26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188"}
Sep 10 19:34:14.525898 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.525825    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd" event={"ID":"63d35b8f-38f0-447b-b60e-2108d67ceb40","Type":"ContainerDied","Data":"a374ca3ac9c3371ae218ad20d77cd8a7cdd02f3b715869c77c334f8eae1fafc1"}
Sep 10 19:34:14.525898 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.525849    2573 scope.go:117] "RemoveContainer" containerID="26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188"
Sep 10 19:34:14.527540 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.527518    2573 generic.go:356] "Generic (PLEG): container finished" podID="0156c21b-afca-4717-920f-62ffff92a259" containerID="ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d" exitCode=0
Sep 10 19:34:14.527658 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.527602    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q" event={"ID":"0156c21b-afca-4717-920f-62ffff92a259","Type":"ContainerDied","Data":"ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d"}
Sep 10 19:34:14.527658 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.527645    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q" event={"ID":"0156c21b-afca-4717-920f-62ffff92a259","Type":"ContainerDied","Data":"99d31d79ebdbcbc0dc62b0886a24fcd63053566aedd8db799f118efdb5ba9efb"}
Sep 10 19:34:14.527797 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.527709    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"
Sep 10 19:34:14.536917 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.536859    2573 scope.go:117] "RemoveContainer" containerID="26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188"
Sep 10 19:34:14.537361 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:34:14.537331    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188\": container with ID starting with 26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188 not found: ID does not exist" containerID="26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188"
Sep 10 19:34:14.537457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.537373    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188"} err="failed to get container status \"26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188\": rpc error: code = NotFound desc = could not find container \"26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188\": container with ID starting with 26411ad07ad7c2bb3e08b7c4831fb2116d7ee6d07a197f3634fdc80d874f9188 not found: ID does not exist"
Sep 10 19:34:14.537457 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.537397    2573 scope.go:117] "RemoveContainer" containerID="ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d"
Sep 10 19:34:14.545964 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.545946    2573 scope.go:117] "RemoveContainer" containerID="ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d"
Sep 10 19:34:14.546281 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:34:14.546261    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d\": container with ID starting with ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d not found: ID does not exist" containerID="ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d"
Sep 10 19:34:14.546337 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.546289    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d"} err="failed to get container status \"ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d\": rpc error: code = NotFound desc = could not find container \"ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d\": container with ID starting with ef15448aa06547a3a5a131140c8263a060b276c9179750f1dde1632a1133497d not found: ID does not exist"
Sep 10 19:34:14.553902 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.553876    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"]
Sep 10 19:34:14.559603 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.559574    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cfa578-6589cdc8f9-jfnhd"]
Sep 10 19:34:14.570959 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.570925    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"]
Sep 10 19:34:14.573506 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.573484    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cfa578-667ccc944f-j984q"]
Sep 10 19:34:14.834066 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.833973    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0156c21b-afca-4717-920f-62ffff92a259" path="/var/lib/kubelet/pods/0156c21b-afca-4717-920f-62ffff92a259/volumes"
Sep 10 19:34:14.834615 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:14.834595    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63d35b8f-38f0-447b-b60e-2108d67ceb40" path="/var/lib/kubelet/pods/63d35b8f-38f0-447b-b60e-2108d67ceb40/volumes"
Sep 10 19:34:24.302266 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302233    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"]
Sep 10 19:34:24.302688 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302674    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63d35b8f-38f0-447b-b60e-2108d67ceb40" containerName="maas-api"
Sep 10 19:34:24.302733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302690    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="63d35b8f-38f0-447b-b60e-2108d67ceb40" containerName="maas-api"
Sep 10 19:34:24.302733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302706    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0156c21b-afca-4717-920f-62ffff92a259" containerName="payload-processing"
Sep 10 19:34:24.302733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302715    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0156c21b-afca-4717-920f-62ffff92a259" containerName="payload-processing"
Sep 10 19:34:24.302733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302725    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ef0f87-d75e-4748-af10-f9d661aad945" containerName="main"
Sep 10 19:34:24.302733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302731    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ef0f87-d75e-4748-af10-f9d661aad945" containerName="main"
Sep 10 19:34:24.302890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302746    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2d030a-7631-48ce-8df7-64fb24e2c008" containerName="test-curl-7528-f52733"
Sep 10 19:34:24.302890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302751    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2d030a-7631-48ce-8df7-64fb24e2c008" containerName="test-curl-7528-f52733"
Sep 10 19:34:24.302890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302765    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff493a16-2e8a-4ac0-969b-f492eca8c5d4" containerName="payload-pre-processing"
Sep 10 19:34:24.302890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302769    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff493a16-2e8a-4ac0-969b-f492eca8c5d4" containerName="payload-pre-processing"
Sep 10 19:34:24.302890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302849    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0156c21b-afca-4717-920f-62ffff92a259" containerName="payload-processing"
Sep 10 19:34:24.302890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302859    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2ef0f87-d75e-4748-af10-f9d661aad945" containerName="main"
Sep 10 19:34:24.302890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302867    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="63d35b8f-38f0-447b-b60e-2108d67ceb40" containerName="maas-api"
Sep 10 19:34:24.302890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302882    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff493a16-2e8a-4ac0-969b-f492eca8c5d4" containerName="payload-pre-processing"
Sep 10 19:34:24.302890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.302887    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2d030a-7631-48ce-8df7-64fb24e2c008" containerName="test-curl-7528-f52733"
Sep 10 19:34:24.305803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.305787    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.310194 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.310169    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"openshift-service-ca.crt\""
Sep 10 19:34:24.311125 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.311108    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"default-dockercfg-rcjfv\""
Sep 10 19:34:24.311213 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.311164    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"rate-test-model-f15c09-kserve-self-signed-certs\""
Sep 10 19:34:24.311213 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.311173    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"kube-root-ca.crt\""
Sep 10 19:34:24.316387 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.316360    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"]
Sep 10 19:34:24.405937 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.405895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-model-cache\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.406122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.405944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e4d868-c581-471b-bcb6-05a4a1efb937-tls-certs\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.406122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.406006    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-dshm\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.406122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.406036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-home\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.406285 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.406135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkxc\" (UniqueName: \"kubernetes.io/projected/67e4d868-c581-471b-bcb6-05a4a1efb937-kube-api-access-qfkxc\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.406285 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.406246    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-tmp-dir\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.507568 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.507532    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-tmp-dir\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.507568 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.507582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-model-cache\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.507825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.507614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e4d868-c581-471b-bcb6-05a4a1efb937-tls-certs\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.507825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.507654    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-dshm\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.507825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.507688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-home\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.507825 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.507754    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkxc\" (UniqueName: \"kubernetes.io/projected/67e4d868-c581-471b-bcb6-05a4a1efb937-kube-api-access-qfkxc\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.508034 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.507959    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-model-cache\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.508034 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.508013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-tmp-dir\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.508177 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.508078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-home\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.510218 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.510191    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-dshm\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.510413 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.510396    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e4d868-c581-471b-bcb6-05a4a1efb937-tls-certs\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.518115 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.518094    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkxc\" (UniqueName: \"kubernetes.io/projected/67e4d868-c581-471b-bcb6-05a4a1efb937-kube-api-access-qfkxc\") pod \"rate-test-model-f15c09-kserve-58b564dc9f-9pcm2\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.615793 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.615689    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:24.888674 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:24.888647    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"]
Sep 10 19:34:24.890270 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:34:24.890243    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67e4d868_c581_471b_bcb6_05a4a1efb937.slice/crio-6518d59d2516e2731de70bd48a2ed78b93542dd2680a7f2de7c6b3fb7cfd154a WatchSource:0}: Error finding container 6518d59d2516e2731de70bd48a2ed78b93542dd2680a7f2de7c6b3fb7cfd154a: Status 404 returned error can't find the container with id 6518d59d2516e2731de70bd48a2ed78b93542dd2680a7f2de7c6b3fb7cfd154a
Sep 10 19:34:25.591748 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:25.591700    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2" event={"ID":"67e4d868-c581-471b-bcb6-05a4a1efb937","Type":"ContainerStarted","Data":"20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9"}
Sep 10 19:34:25.591748 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:25.591749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2" event={"ID":"67e4d868-c581-471b-bcb6-05a4a1efb937","Type":"ContainerStarted","Data":"6518d59d2516e2731de70bd48a2ed78b93542dd2680a7f2de7c6b3fb7cfd154a"}
Sep 10 19:34:25.609705 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:25.609654    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2" podStartSLOduration=1.609637988 podStartE2EDuration="1.609637988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:25.608858998 +0000 UTC m=+1401.268656210" watchObservedRunningTime="2026-09-10 19:34:25.609637988 +0000 UTC m=+1401.269435180"
Sep 10 19:34:32.966014 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:32.965972    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"]
Sep 10 19:34:32.966496 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:32.966194    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb" podUID="2795cad8-c5d4-4694-8a22-0a17a2e2c73e" containerName="maas-api" containerID="cri-o://819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8" gracePeriod=30
Sep 10 19:34:33.114835 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.114784    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"]
Sep 10 19:34:33.115658 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.115624    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7" podUID="0a190b42-a310-43a2-a93d-6f95a298f19c" containerName="payload-pre-processing" containerID="cri-o://d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26" gracePeriod=30
Sep 10 19:34:33.242686 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.242659    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:34:33.363353 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.363327    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:34:33.398952 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.398911    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-maas-api-tls\") pod \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") "
Sep 10 19:34:33.399135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.398999    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-openshift-service-ca\") pod \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") "
Sep 10 19:34:33.399135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.399017    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-metrics-certs\") pod \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") "
Sep 10 19:34:33.399135 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.399064    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn2ws\" (UniqueName: \"kubernetes.io/projected/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-kube-api-access-zn2ws\") pod \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\" (UID: \"2795cad8-c5d4-4694-8a22-0a17a2e2c73e\") "
Sep 10 19:34:33.399459 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.399434    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2795cad8-c5d4-4694-8a22-0a17a2e2c73e" (UID: "2795cad8-c5d4-4694-8a22-0a17a2e2c73e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:34:33.401315 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.401289    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2795cad8-c5d4-4694-8a22-0a17a2e2c73e" (UID: "2795cad8-c5d4-4694-8a22-0a17a2e2c73e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:34:33.401441 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.401417    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2795cad8-c5d4-4694-8a22-0a17a2e2c73e" (UID: "2795cad8-c5d4-4694-8a22-0a17a2e2c73e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:34:33.401489 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.401465    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-kube-api-access-zn2ws" (OuterVolumeSpecName: "kube-api-access-zn2ws") pod "2795cad8-c5d4-4694-8a22-0a17a2e2c73e" (UID: "2795cad8-c5d4-4694-8a22-0a17a2e2c73e"). InnerVolumeSpecName "kube-api-access-zn2ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:34:33.500064 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.499978    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhfvm\" (UniqueName: \"kubernetes.io/projected/0a190b42-a310-43a2-a93d-6f95a298f19c-kube-api-access-nhfvm\") pod \"0a190b42-a310-43a2-a93d-6f95a298f19c\" (UID: \"0a190b42-a310-43a2-a93d-6f95a298f19c\") "
Sep 10 19:34:33.500247 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.500172    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a190b42-a310-43a2-a93d-6f95a298f19c-plugins-config-volume\") pod \"0a190b42-a310-43a2-a93d-6f95a298f19c\" (UID: \"0a190b42-a310-43a2-a93d-6f95a298f19c\") "
Sep 10 19:34:33.500496 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.500476    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:33.500556 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.500501    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:33.500556 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.500516    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2ws\" (UniqueName: \"kubernetes.io/projected/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-kube-api-access-zn2ws\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:33.500556 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.500530    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2795cad8-c5d4-4694-8a22-0a17a2e2c73e-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:33.500556 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.500533    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a190b42-a310-43a2-a93d-6f95a298f19c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a190b42-a310-43a2-a93d-6f95a298f19c" (UID: "0a190b42-a310-43a2-a93d-6f95a298f19c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:34:33.502210 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.502186    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a190b42-a310-43a2-a93d-6f95a298f19c-kube-api-access-nhfvm" (OuterVolumeSpecName: "kube-api-access-nhfvm") pod "0a190b42-a310-43a2-a93d-6f95a298f19c" (UID: "0a190b42-a310-43a2-a93d-6f95a298f19c"). InnerVolumeSpecName "kube-api-access-nhfvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:34:33.601327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.601292    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhfvm\" (UniqueName: \"kubernetes.io/projected/0a190b42-a310-43a2-a93d-6f95a298f19c-kube-api-access-nhfvm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:33.601327 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.601320    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a190b42-a310-43a2-a93d-6f95a298f19c-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:34:33.638771 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.638728    2573 generic.go:356] "Generic (PLEG): container finished" podID="2795cad8-c5d4-4694-8a22-0a17a2e2c73e" containerID="819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8" exitCode=0
Sep 10 19:34:33.638965 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.638778    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"
Sep 10 19:34:33.638965 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.638827    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb" event={"ID":"2795cad8-c5d4-4694-8a22-0a17a2e2c73e","Type":"ContainerDied","Data":"819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8"}
Sep 10 19:34:33.638965 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.638880    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb" event={"ID":"2795cad8-c5d4-4694-8a22-0a17a2e2c73e","Type":"ContainerDied","Data":"176523d1da7e13b825029e674efcbfce8f08f652e378086b10a81d04dad0c6ec"}
Sep 10 19:34:33.638965 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.638908    2573 scope.go:117] "RemoveContainer" containerID="819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8"
Sep 10 19:34:33.640620 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.640597    2573 generic.go:356] "Generic (PLEG): container finished" podID="0a190b42-a310-43a2-a93d-6f95a298f19c" containerID="d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26" exitCode=0
Sep 10 19:34:33.640731 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.640646    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7" event={"ID":"0a190b42-a310-43a2-a93d-6f95a298f19c","Type":"ContainerDied","Data":"d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26"}
Sep 10 19:34:33.640731 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.640691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7" event={"ID":"0a190b42-a310-43a2-a93d-6f95a298f19c","Type":"ContainerDied","Data":"ae12054180de3e8eab4f3f4e006e317d4d1676e2f91b57e2c0f223a868343633"}
Sep 10 19:34:33.640731 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.640652    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"
Sep 10 19:34:33.650023 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.650001    2573 scope.go:117] "RemoveContainer" containerID="819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8"
Sep 10 19:34:33.650410 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:34:33.650389    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8\": container with ID starting with 819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8 not found: ID does not exist" containerID="819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8"
Sep 10 19:34:33.650518 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.650418    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8"} err="failed to get container status \"819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8\": rpc error: code = NotFound desc = could not find container \"819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8\": container with ID starting with 819a374a27df5d83424506e0cf73c26682166b7629cff2941710d3a526bbbfa8 not found: ID does not exist"
Sep 10 19:34:33.650518 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.650440    2573 scope.go:117] "RemoveContainer" containerID="d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26"
Sep 10 19:34:33.659201 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.659179    2573 scope.go:117] "RemoveContainer" containerID="d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26"
Sep 10 19:34:33.659533 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:34:33.659494    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26\": container with ID starting with d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26 not found: ID does not exist" containerID="d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26"
Sep 10 19:34:33.659635 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.659540    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26"} err="failed to get container status \"d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26\": rpc error: code = NotFound desc = could not find container \"d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26\": container with ID starting with d53b0f855d0595dc06057750d4f7ca547465c09f1d47525fe3b94d8c07dbcd26 not found: ID does not exist"
Sep 10 19:34:33.687068 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.687038    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"]
Sep 10 19:34:33.689694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.689662    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-94ca92-85d66ff69d-lrtp7"]
Sep 10 19:34:33.704524 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.704500    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"]
Sep 10 19:34:33.710720 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:33.710696    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-94ca92-8448d5bd66-blvlb"]
Sep 10 19:34:34.616775 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:34.616740    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:34.617180 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:34.616789    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:34.629333 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:34.629306    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:34.659260 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:34.659230    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:34:34.834864 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:34.834830    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a190b42-a310-43a2-a93d-6f95a298f19c" path="/var/lib/kubelet/pods/0a190b42-a310-43a2-a93d-6f95a298f19c/volumes"
Sep 10 19:34:34.835501 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:34.835483    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2795cad8-c5d4-4694-8a22-0a17a2e2c73e" path="/var/lib/kubelet/pods/2795cad8-c5d4-4694-8a22-0a17a2e2c73e/volumes"
Sep 10 19:34:59.813350 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:34:59.813319    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"]
Sep 10 19:35:04.616743 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.616710    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"]
Sep 10 19:35:04.617278 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.617118    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a190b42-a310-43a2-a93d-6f95a298f19c" containerName="payload-pre-processing"
Sep 10 19:35:04.617278 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.617128    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a190b42-a310-43a2-a93d-6f95a298f19c" containerName="payload-pre-processing"
Sep 10 19:35:04.617278 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.617144    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2795cad8-c5d4-4694-8a22-0a17a2e2c73e" containerName="maas-api"
Sep 10 19:35:04.617278 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.617169    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2795cad8-c5d4-4694-8a22-0a17a2e2c73e" containerName="maas-api"
Sep 10 19:35:04.617278 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.617235    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2795cad8-c5d4-4694-8a22-0a17a2e2c73e" containerName="maas-api"
Sep 10 19:35:04.617278 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.617249    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a190b42-a310-43a2-a93d-6f95a298f19c" containerName="payload-pre-processing"
Sep 10 19:35:04.620390 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.620370    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.623079 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.623051    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c59b36-serving-cert\""
Sep 10 19:35:04.633695 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.633666    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"]
Sep 10 19:35:04.702206 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.702139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b970999-758d-4c05-be9d-ab87cf1855d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.702411 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.702261    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.702411 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.702339    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fms7g\" (UniqueName: \"kubernetes.io/projected/9b970999-758d-4c05-be9d-ab87cf1855d8-kube-api-access-fms7g\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.702411 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.702387    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-metrics-certs\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.803410 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.803373    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.803613 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.803457    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fms7g\" (UniqueName: \"kubernetes.io/projected/9b970999-758d-4c05-be9d-ab87cf1855d8-kube-api-access-fms7g\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.803613 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.803505    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-metrics-certs\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.803613 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.803567    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b970999-758d-4c05-be9d-ab87cf1855d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.804329 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.804304    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b970999-758d-4c05-be9d-ab87cf1855d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.806066 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.806042    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c59b36-serving-cert\""
Sep 10 19:35:04.806363 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.806344    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-metrics-certs\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.811973 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:04.811950    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fms7g\" (UniqueName: \"kubernetes.io/projected/9b970999-758d-4c05-be9d-ab87cf1855d8-kube-api-access-fms7g\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:04.813727 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:35:04.813709    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-serving-cert: secret "maas-api-e2e-ipp-a-c59b36-serving-cert" not found
Sep 10 19:35:04.813805 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:35:04.813781    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-maas-api-tls podName:9b970999-758d-4c05-be9d-ab87cf1855d8 nodeName:}" failed. No retries permitted until 2026-09-10 19:35:05.313760603 +0000 UTC m=+1440.973557792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-maas-api-tls") pod "maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq" (UID: "9b970999-758d-4c05-be9d-ab87cf1855d8") : secret "maas-api-e2e-ipp-a-c59b36-serving-cert" not found
Sep 10 19:35:05.409880 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.409788    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:05.412389 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.412370    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:05.531769 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.531726    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:05.572364 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.572329    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"]
Sep 10 19:35:05.576772 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.576746    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:05.579727 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.579706    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c59b36-dockercfg-fb7ds\""
Sep 10 19:35:05.579860 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.579800    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c59b36\""
Sep 10 19:35:05.585021 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.584998    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"]
Sep 10 19:35:05.712926 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.712839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3846470e-e119-4e04-a3e4-93d372f5ca56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w\" (UID: \"3846470e-e119-4e04-a3e4-93d372f5ca56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:05.713319 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.712976    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmfb\" (UniqueName: \"kubernetes.io/projected/3846470e-e119-4e04-a3e4-93d372f5ca56-kube-api-access-kbmfb\") pod \"payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w\" (UID: \"3846470e-e119-4e04-a3e4-93d372f5ca56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:05.816243 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.816097    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"]
Sep 10 19:35:05.823496 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.817200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3846470e-e119-4e04-a3e4-93d372f5ca56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w\" (UID: \"3846470e-e119-4e04-a3e4-93d372f5ca56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:05.823496 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.817369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmfb\" (UniqueName: \"kubernetes.io/projected/3846470e-e119-4e04-a3e4-93d372f5ca56-kube-api-access-kbmfb\") pod \"payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w\" (UID: \"3846470e-e119-4e04-a3e4-93d372f5ca56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:05.823496 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.818050    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3846470e-e119-4e04-a3e4-93d372f5ca56-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w\" (UID: \"3846470e-e119-4e04-a3e4-93d372f5ca56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:05.824670 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.824647    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:35:05.827074 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.827049    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmfb\" (UniqueName: \"kubernetes.io/projected/3846470e-e119-4e04-a3e4-93d372f5ca56-kube-api-access-kbmfb\") pod \"payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w\" (UID: \"3846470e-e119-4e04-a3e4-93d372f5ca56\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:05.892830 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:05.892785    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:06.171381 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:06.171357    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"]
Sep 10 19:35:06.173187 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:35:06.173139    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3846470e_e119_4e04_a3e4_93d372f5ca56.slice/crio-54b1702794796d408f48d9e3a8d8602f9df142712aeff71afd81bfc842db2ea0 WatchSource:0}: Error finding container 54b1702794796d408f48d9e3a8d8602f9df142712aeff71afd81bfc842db2ea0: Status 404 returned error can't find the container with id 54b1702794796d408f48d9e3a8d8602f9df142712aeff71afd81bfc842db2ea0
Sep 10 19:35:06.819545 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:06.819508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq" event={"ID":"9b970999-758d-4c05-be9d-ab87cf1855d8","Type":"ContainerStarted","Data":"e31819cf326f40513c417b5ae2ff66cbf2d108b195b484e21223d27e7a50ba6b"}
Sep 10 19:35:06.820024 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:06.819573    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq" event={"ID":"9b970999-758d-4c05-be9d-ab87cf1855d8","Type":"ContainerStarted","Data":"02008cb28a36653be66a789a79b24046ca6178b5c75208e3520d71602df51964"}
Sep 10 19:35:06.820024 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:06.819715    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:06.821233 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:06.821211    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w" event={"ID":"3846470e-e119-4e04-a3e4-93d372f5ca56","Type":"ContainerStarted","Data":"fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2"}
Sep 10 19:35:06.821366 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:06.821250    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w" event={"ID":"3846470e-e119-4e04-a3e4-93d372f5ca56","Type":"ContainerStarted","Data":"54b1702794796d408f48d9e3a8d8602f9df142712aeff71afd81bfc842db2ea0"}
Sep 10 19:35:06.821433 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:06.821362    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:06.837602 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:06.837466    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq" podStartSLOduration=2.247399182 podStartE2EDuration="2.837446791s" totalImagesPullingTime="590.047609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:06.836780583 +0000 UTC m=+1442.496577779" watchObservedRunningTime="2026-09-10 19:35:06.837446791 +0000 UTC m=+1442.497243991"
Sep 10 19:35:06.853553 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:06.853509    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w" podStartSLOduration=1.853494427 podStartE2EDuration="1.853494427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:06.852858219 +0000 UTC m=+1442.512655413" watchObservedRunningTime="2026-09-10 19:35:06.853494427 +0000 UTC m=+1442.513291619"
Sep 10 19:35:12.834804 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:12.834774    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:35:14.833980 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:14.833928    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2" podUID="67e4d868-c581-471b-bcb6-05a4a1efb937" containerName="main" containerID="cri-o://20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9" gracePeriod=45
Sep 10 19:35:15.078132 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.078104    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:35:15.116745 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.116662    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-model-cache\") pod \"67e4d868-c581-471b-bcb6-05a4a1efb937\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") "
Sep 10 19:35:15.116896 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.116759    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-tmp-dir\") pod \"67e4d868-c581-471b-bcb6-05a4a1efb937\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") "
Sep 10 19:35:15.116896 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.116796    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-home\") pod \"67e4d868-c581-471b-bcb6-05a4a1efb937\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") "
Sep 10 19:35:15.116896 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.116832    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfkxc\" (UniqueName: \"kubernetes.io/projected/67e4d868-c581-471b-bcb6-05a4a1efb937-kube-api-access-qfkxc\") pod \"67e4d868-c581-471b-bcb6-05a4a1efb937\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") "
Sep 10 19:35:15.117064 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.116900    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e4d868-c581-471b-bcb6-05a4a1efb937-tls-certs\") pod \"67e4d868-c581-471b-bcb6-05a4a1efb937\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") "
Sep 10 19:35:15.117064 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.116933    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-dshm\") pod \"67e4d868-c581-471b-bcb6-05a4a1efb937\" (UID: \"67e4d868-c581-471b-bcb6-05a4a1efb937\") "
Sep 10 19:35:15.117064 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.117031    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-model-cache" (OuterVolumeSpecName: "model-cache") pod "67e4d868-c581-471b-bcb6-05a4a1efb937" (UID: "67e4d868-c581-471b-bcb6-05a4a1efb937"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:35:15.117266 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.117071    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67e4d868-c581-471b-bcb6-05a4a1efb937" (UID: "67e4d868-c581-471b-bcb6-05a4a1efb937"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:35:15.117266 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.117082    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-home" (OuterVolumeSpecName: "home") pod "67e4d868-c581-471b-bcb6-05a4a1efb937" (UID: "67e4d868-c581-471b-bcb6-05a4a1efb937"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:35:15.117371 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.117342    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:35:15.117371 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.117363    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:35:15.117472 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.117376    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:35:15.119342 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.119308    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-dshm" (OuterVolumeSpecName: "dshm") pod "67e4d868-c581-471b-bcb6-05a4a1efb937" (UID: "67e4d868-c581-471b-bcb6-05a4a1efb937"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:35:15.119444 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.119376    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e4d868-c581-471b-bcb6-05a4a1efb937-kube-api-access-qfkxc" (OuterVolumeSpecName: "kube-api-access-qfkxc") pod "67e4d868-c581-471b-bcb6-05a4a1efb937" (UID: "67e4d868-c581-471b-bcb6-05a4a1efb937"). InnerVolumeSpecName "kube-api-access-qfkxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:35:15.119611 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.119589    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e4d868-c581-471b-bcb6-05a4a1efb937-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67e4d868-c581-471b-bcb6-05a4a1efb937" (UID: "67e4d868-c581-471b-bcb6-05a4a1efb937"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:35:15.217914 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.217884    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkxc\" (UniqueName: \"kubernetes.io/projected/67e4d868-c581-471b-bcb6-05a4a1efb937-kube-api-access-qfkxc\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:35:15.217914 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.217914    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e4d868-c581-471b-bcb6-05a4a1efb937-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:35:15.218198 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.217930    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e4d868-c581-471b-bcb6-05a4a1efb937-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:35:15.868647 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.868608    2573 generic.go:356] "Generic (PLEG): container finished" podID="67e4d868-c581-471b-bcb6-05a4a1efb937" containerID="20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9" exitCode=0
Sep 10 19:35:15.869075 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.868659    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"
Sep 10 19:35:15.869075 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.868661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2" event={"ID":"67e4d868-c581-471b-bcb6-05a4a1efb937","Type":"ContainerDied","Data":"20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9"}
Sep 10 19:35:15.869075 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.868704    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2" event={"ID":"67e4d868-c581-471b-bcb6-05a4a1efb937","Type":"ContainerDied","Data":"6518d59d2516e2731de70bd48a2ed78b93542dd2680a7f2de7c6b3fb7cfd154a"}
Sep 10 19:35:15.869075 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.868727    2573 scope.go:117] "RemoveContainer" containerID="20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9"
Sep 10 19:35:15.878467 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.878447    2573 scope.go:117] "RemoveContainer" containerID="20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9"
Sep 10 19:35:15.878749 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:35:15.878728    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9\": container with ID starting with 20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9 not found: ID does not exist" containerID="20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9"
Sep 10 19:35:15.878800 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.878758    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9"} err="failed to get container status \"20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9\": rpc error: code = NotFound desc = could not find container \"20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9\": container with ID starting with 20b89c6a756b1c15c83f1133379306a3b8fd697428875af2fd8d6e08804644e9 not found: ID does not exist"
Sep 10 19:35:15.890840 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.890808    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"]
Sep 10 19:35:15.909378 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:15.909352    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/rate-test-model-f15c09-kserve-58b564dc9f-9pcm2"]
Sep 10 19:35:16.834484 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:16.834447    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e4d868-c581-471b-bcb6-05a4a1efb937" path="/var/lib/kubelet/pods/67e4d868-c581-471b-bcb6-05a4a1efb937/volumes"
Sep 10 19:35:17.829108 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:17.829075    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:35:34.586826 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.586785    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"]
Sep 10 19:35:34.587283 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.587214    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e4d868-c581-471b-bcb6-05a4a1efb937" containerName="main"
Sep 10 19:35:34.587283 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.587226    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e4d868-c581-471b-bcb6-05a4a1efb937" containerName="main"
Sep 10 19:35:34.587365 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.587305    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e4d868-c581-471b-bcb6-05a4a1efb937" containerName="main"
Sep 10 19:35:34.590644 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.590623    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:35:34.593475 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.593459    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-90ceb3-dockercfg-jmqqp\""
Sep 10 19:35:34.593555 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.593489    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-90ceb3\""
Sep 10 19:35:34.605453 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.605428    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"]
Sep 10 19:35:34.708241 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.708208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99fcb4e3-f28c-420c-98e4-07484485fc51-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4\" (UID: \"99fcb4e3-f28c-420c-98e4-07484485fc51\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:35:34.708409 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.708294    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2kc\" (UniqueName: \"kubernetes.io/projected/99fcb4e3-f28c-420c-98e4-07484485fc51-kube-api-access-cv2kc\") pod \"payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4\" (UID: \"99fcb4e3-f28c-420c-98e4-07484485fc51\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:35:34.809436 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.809397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99fcb4e3-f28c-420c-98e4-07484485fc51-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4\" (UID: \"99fcb4e3-f28c-420c-98e4-07484485fc51\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:35:34.809614 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.809454    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2kc\" (UniqueName: \"kubernetes.io/projected/99fcb4e3-f28c-420c-98e4-07484485fc51-kube-api-access-cv2kc\") pod \"payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4\" (UID: \"99fcb4e3-f28c-420c-98e4-07484485fc51\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:35:34.810192 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.810138    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99fcb4e3-f28c-420c-98e4-07484485fc51-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4\" (UID: \"99fcb4e3-f28c-420c-98e4-07484485fc51\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:35:34.820635 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.820608    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2kc\" (UniqueName: \"kubernetes.io/projected/99fcb4e3-f28c-420c-98e4-07484485fc51-kube-api-access-cv2kc\") pod \"payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4\" (UID: \"99fcb4e3-f28c-420c-98e4-07484485fc51\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:35:34.900963 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:34.900863    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:35:35.175704 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:35.175674    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"]
Sep 10 19:35:35.178108 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:35:35.178081    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fcb4e3_f28c_420c_98e4_07484485fc51.slice/crio-25c8665bb238a37bcd8559bc0dfb22368b5d5b100cd2a2257580b79cb1bc86c2 WatchSource:0}: Error finding container 25c8665bb238a37bcd8559bc0dfb22368b5d5b100cd2a2257580b79cb1bc86c2: Status 404 returned error can't find the container with id 25c8665bb238a37bcd8559bc0dfb22368b5d5b100cd2a2257580b79cb1bc86c2
Sep 10 19:35:35.975107 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:35.975065    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4" event={"ID":"99fcb4e3-f28c-420c-98e4-07484485fc51","Type":"ContainerStarted","Data":"f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569"}
Sep 10 19:35:35.975607 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:35.975123    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4" event={"ID":"99fcb4e3-f28c-420c-98e4-07484485fc51","Type":"ContainerStarted","Data":"25c8665bb238a37bcd8559bc0dfb22368b5d5b100cd2a2257580b79cb1bc86c2"}
Sep 10 19:35:35.975607 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:35.975196    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:35:35.992591 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:35.992536    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4" podStartSLOduration=1.9925202720000001 podStartE2EDuration="1.992520272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:35.992334622 +0000 UTC m=+1471.652131838" watchObservedRunningTime="2026-09-10 19:35:35.992520272 +0000 UTC m=+1471.652317466"
Sep 10 19:35:46.982276 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:35:46.982245    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:36:02.661706 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:02.661670    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4_99fcb4e3-f28c-420c-98e4-07484485fc51/payload-processing/0.log"
Sep 10 19:36:33.221624 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:33.221597    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w_3846470e-e119-4e04-a3e4-93d372f5ca56/payload-processing/0.log"
Sep 10 19:36:33.460888 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:33.460858    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4_99fcb4e3-f28c-420c-98e4-07484485fc51/payload-processing/0.log"
Sep 10 19:36:37.776514 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:37.776477    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"]
Sep 10 19:36:37.780318 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:37.780295    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:37.782704 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:37.782684    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-cbd9f0-serving-cert\""
Sep 10 19:36:37.792464 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:37.792438    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"]
Sep 10 19:36:37.937424 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:37.937391    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:37.937637 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:37.937451    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpcv\" (UniqueName: \"kubernetes.io/projected/0683e501-1727-482e-a54b-750606d43e5f-kube-api-access-vzpcv\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:37.937637 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:37.937549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:37.937637 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:37.937579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683e501-1727-482e-a54b-750606d43e5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.038991 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.038885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.038991 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.038951    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpcv\" (UniqueName: \"kubernetes.io/projected/0683e501-1727-482e-a54b-750606d43e5f-kube-api-access-vzpcv\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.039291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.039023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.039291 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.039054    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683e501-1727-482e-a54b-750606d43e5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.039731 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.039697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683e501-1727-482e-a54b-750606d43e5f-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.041661 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.041638    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.041738 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.041721    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.048015 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.047993    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpcv\" (UniqueName: \"kubernetes.io/projected/0683e501-1727-482e-a54b-750606d43e5f-kube-api-access-vzpcv\") pod \"maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.093601 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.093563    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:38.373215 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.373191    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"]
Sep 10 19:36:38.375008 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:36:38.374982    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0683e501_1727_482e_a54b_750606d43e5f.slice/crio-073d8f3a6e1069ed11b8d4351e6e32b9e2f4f15a1947b8e7a61b58526cd89804 WatchSource:0}: Error finding container 073d8f3a6e1069ed11b8d4351e6e32b9e2f4f15a1947b8e7a61b58526cd89804: Status 404 returned error can't find the container with id 073d8f3a6e1069ed11b8d4351e6e32b9e2f4f15a1947b8e7a61b58526cd89804
Sep 10 19:36:38.421811 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.421775    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"]
Sep 10 19:36:38.426094 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.426077    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:36:38.429314 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.429295    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-cbd9f0\""
Sep 10 19:36:38.436178 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.436140    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"]
Sep 10 19:36:38.542735 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.542690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7z7v\" (UniqueName: \"kubernetes.io/projected/c2a949a4-2aa0-466e-b340-02b3e5fcb408-kube-api-access-q7z7v\") pod \"payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r\" (UID: \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:36:38.542915 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.542769    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a949a4-2aa0-466e-b340-02b3e5fcb408-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r\" (UID: \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:36:38.644403 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.644323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a949a4-2aa0-466e-b340-02b3e5fcb408-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r\" (UID: \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:36:38.644554 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.644424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7z7v\" (UniqueName: \"kubernetes.io/projected/c2a949a4-2aa0-466e-b340-02b3e5fcb408-kube-api-access-q7z7v\") pod \"payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r\" (UID: \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:36:38.645024 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.645002    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a949a4-2aa0-466e-b340-02b3e5fcb408-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r\" (UID: \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:36:38.653816 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.653787    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7z7v\" (UniqueName: \"kubernetes.io/projected/c2a949a4-2aa0-466e-b340-02b3e5fcb408-kube-api-access-q7z7v\") pod \"payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r\" (UID: \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:36:38.737881 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:38.737844    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:36:39.014374 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:39.014349    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"]
Sep 10 19:36:39.016728 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:36:39.016665    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a949a4_2aa0_466e_b340_02b3e5fcb408.slice/crio-435600ae602b2dc1f2bb7a9f4bcf8f353de0f340a513e3ab94eeeecb1aeb7ab3 WatchSource:0}: Error finding container 435600ae602b2dc1f2bb7a9f4bcf8f353de0f340a513e3ab94eeeecb1aeb7ab3: Status 404 returned error can't find the container with id 435600ae602b2dc1f2bb7a9f4bcf8f353de0f340a513e3ab94eeeecb1aeb7ab3
Sep 10 19:36:39.314267 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:39.314227    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r" event={"ID":"c2a949a4-2aa0-466e-b340-02b3e5fcb408","Type":"ContainerStarted","Data":"1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9"}
Sep 10 19:36:39.314480 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:39.314318    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r" event={"ID":"c2a949a4-2aa0-466e-b340-02b3e5fcb408","Type":"ContainerStarted","Data":"435600ae602b2dc1f2bb7a9f4bcf8f353de0f340a513e3ab94eeeecb1aeb7ab3"}
Sep 10 19:36:39.314480 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:39.314353    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:36:39.316015 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:39.315990    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5" event={"ID":"0683e501-1727-482e-a54b-750606d43e5f","Type":"ContainerStarted","Data":"37550af78f8e1c535eec33ef0668c2253121a90c9fe45c5afc74210672865d63"}
Sep 10 19:36:39.316118 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:39.316032    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5" event={"ID":"0683e501-1727-482e-a54b-750606d43e5f","Type":"ContainerStarted","Data":"073d8f3a6e1069ed11b8d4351e6e32b9e2f4f15a1947b8e7a61b58526cd89804"}
Sep 10 19:36:39.316118 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:39.316063    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:39.335817 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:39.335754    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r" podStartSLOduration=1.335736478 podStartE2EDuration="1.335736478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:36:39.334712286 +0000 UTC m=+1534.994509476" watchObservedRunningTime="2026-09-10 19:36:39.335736478 +0000 UTC m=+1534.995533684"
Sep 10 19:36:45.327258 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:45.327224    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:36:45.347960 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:45.347902    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5" podStartSLOduration=7.726748215 podStartE2EDuration="8.347886243s" totalImagesPullingTime="621.138028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:36:39.354084407 +0000 UTC m=+1535.013881600" watchObservedRunningTime="2026-09-10 19:36:45.347886243 +0000 UTC m=+1541.007683468"
Sep 10 19:36:50.323791 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:36:50.323755    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:37:08.336195 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.336160    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"]
Sep 10 19:37:08.336775 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.336424    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5" podUID="0683e501-1727-482e-a54b-750606d43e5f" containerName="maas-api" containerID="cri-o://37550af78f8e1c535eec33ef0668c2253121a90c9fe45c5afc74210672865d63" gracePeriod=30
Sep 10 19:37:08.410388 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.410342    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"]
Sep 10 19:37:08.410865 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.410828    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r" podUID="c2a949a4-2aa0-466e-b340-02b3e5fcb408" containerName="payload-pre-processing" containerID="cri-o://1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9" gracePeriod=30
Sep 10 19:37:08.473850 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.473820    2573 generic.go:356] "Generic (PLEG): container finished" podID="0683e501-1727-482e-a54b-750606d43e5f" containerID="37550af78f8e1c535eec33ef0668c2253121a90c9fe45c5afc74210672865d63" exitCode=0
Sep 10 19:37:08.473991 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.473862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5" event={"ID":"0683e501-1727-482e-a54b-750606d43e5f","Type":"ContainerDied","Data":"37550af78f8e1c535eec33ef0668c2253121a90c9fe45c5afc74210672865d63"}
Sep 10 19:37:08.658243 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.658210    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:37:08.689693 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.689661    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:37:08.747030 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.746996    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-metrics-certs\") pod \"0683e501-1727-482e-a54b-750606d43e5f\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") "
Sep 10 19:37:08.747240 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.747055    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-maas-api-tls\") pod \"0683e501-1727-482e-a54b-750606d43e5f\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") "
Sep 10 19:37:08.747240 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.747097    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzpcv\" (UniqueName: \"kubernetes.io/projected/0683e501-1727-482e-a54b-750606d43e5f-kube-api-access-vzpcv\") pod \"0683e501-1727-482e-a54b-750606d43e5f\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") "
Sep 10 19:37:08.747240 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.747181    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a949a4-2aa0-466e-b340-02b3e5fcb408-plugins-config-volume\") pod \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\" (UID: \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\") "
Sep 10 19:37:08.747433 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.747260    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683e501-1727-482e-a54b-750606d43e5f-openshift-service-ca\") pod \"0683e501-1727-482e-a54b-750606d43e5f\" (UID: \"0683e501-1727-482e-a54b-750606d43e5f\") "
Sep 10 19:37:08.747433 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.747349    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7z7v\" (UniqueName: \"kubernetes.io/projected/c2a949a4-2aa0-466e-b340-02b3e5fcb408-kube-api-access-q7z7v\") pod \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\" (UID: \"c2a949a4-2aa0-466e-b340-02b3e5fcb408\") "
Sep 10 19:37:08.747604 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.747570    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a949a4-2aa0-466e-b340-02b3e5fcb408-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2a949a4-2aa0-466e-b340-02b3e5fcb408" (UID: "c2a949a4-2aa0-466e-b340-02b3e5fcb408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:37:08.747675 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.747649    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0683e501-1727-482e-a54b-750606d43e5f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0683e501-1727-482e-a54b-750606d43e5f" (UID: "0683e501-1727-482e-a54b-750606d43e5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:37:08.747773 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.747755    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2a949a4-2aa0-466e-b340-02b3e5fcb408-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:08.747998 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.747780    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0683e501-1727-482e-a54b-750606d43e5f-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:08.749455 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.749429    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0683e501-1727-482e-a54b-750606d43e5f-kube-api-access-vzpcv" (OuterVolumeSpecName: "kube-api-access-vzpcv") pod "0683e501-1727-482e-a54b-750606d43e5f" (UID: "0683e501-1727-482e-a54b-750606d43e5f"). InnerVolumeSpecName "kube-api-access-vzpcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:37:08.749559 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.749522    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0683e501-1727-482e-a54b-750606d43e5f" (UID: "0683e501-1727-482e-a54b-750606d43e5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:37:08.749944 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.749923    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0683e501-1727-482e-a54b-750606d43e5f" (UID: "0683e501-1727-482e-a54b-750606d43e5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:37:08.749995 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.749936    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a949a4-2aa0-466e-b340-02b3e5fcb408-kube-api-access-q7z7v" (OuterVolumeSpecName: "kube-api-access-q7z7v") pod "c2a949a4-2aa0-466e-b340-02b3e5fcb408" (UID: "c2a949a4-2aa0-466e-b340-02b3e5fcb408"). InnerVolumeSpecName "kube-api-access-q7z7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:37:08.848668 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.848643    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7z7v\" (UniqueName: \"kubernetes.io/projected/c2a949a4-2aa0-466e-b340-02b3e5fcb408-kube-api-access-q7z7v\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:08.848668 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.848668    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:08.848806 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.848677    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0683e501-1727-482e-a54b-750606d43e5f-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:08.848806 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:08.848685    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzpcv\" (UniqueName: \"kubernetes.io/projected/0683e501-1727-482e-a54b-750606d43e5f-kube-api-access-vzpcv\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:09.480006 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.479972    2573 generic.go:356] "Generic (PLEG): container finished" podID="c2a949a4-2aa0-466e-b340-02b3e5fcb408" containerID="1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9" exitCode=0
Sep 10 19:37:09.480519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.480021    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"
Sep 10 19:37:09.480519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.480038    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r" event={"ID":"c2a949a4-2aa0-466e-b340-02b3e5fcb408","Type":"ContainerDied","Data":"1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9"}
Sep 10 19:37:09.480519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.480083    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r" event={"ID":"c2a949a4-2aa0-466e-b340-02b3e5fcb408","Type":"ContainerDied","Data":"435600ae602b2dc1f2bb7a9f4bcf8f353de0f340a513e3ab94eeeecb1aeb7ab3"}
Sep 10 19:37:09.480519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.480112    2573 scope.go:117] "RemoveContainer" containerID="1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9"
Sep 10 19:37:09.481828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.481805    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"
Sep 10 19:37:09.481940 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.481816    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5" event={"ID":"0683e501-1727-482e-a54b-750606d43e5f","Type":"ContainerDied","Data":"073d8f3a6e1069ed11b8d4351e6e32b9e2f4f15a1947b8e7a61b58526cd89804"}
Sep 10 19:37:09.489033 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.489016    2573 scope.go:117] "RemoveContainer" containerID="1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9"
Sep 10 19:37:09.489325 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:37:09.489303    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9\": container with ID starting with 1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9 not found: ID does not exist" containerID="1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9"
Sep 10 19:37:09.489418 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.489330    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9"} err="failed to get container status \"1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9\": rpc error: code = NotFound desc = could not find container \"1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9\": container with ID starting with 1e98c3b9f3f51a4f4e6afa1332017772701103e0f69a216914a60ab67511ccf9 not found: ID does not exist"
Sep 10 19:37:09.489418 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.489347    2573 scope.go:117] "RemoveContainer" containerID="37550af78f8e1c535eec33ef0668c2253121a90c9fe45c5afc74210672865d63"
Sep 10 19:37:09.499020 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.499001    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"]
Sep 10 19:37:09.506143 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.506118    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cbd9f0-5b76887ccd-l9n4r"]
Sep 10 19:37:09.520350 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.520324    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"]
Sep 10 19:37:09.525157 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:09.525128    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-cbd9f0-74f59957d6-p5vr5"]
Sep 10 19:37:10.835183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:10.835133    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0683e501-1727-482e-a54b-750606d43e5f" path="/var/lib/kubelet/pods/0683e501-1727-482e-a54b-750606d43e5f/volumes"
Sep 10 19:37:10.835568 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:10.835535    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a949a4-2aa0-466e-b340-02b3e5fcb408" path="/var/lib/kubelet/pods/c2a949a4-2aa0-466e-b340-02b3e5fcb408/volumes"
Sep 10 19:37:34.461743 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.461708    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"]
Sep 10 19:37:34.464533 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.461965    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq" podUID="9b970999-758d-4c05-be9d-ab87cf1855d8" containerName="maas-api" containerID="cri-o://e31819cf326f40513c417b5ae2ff66cbf2d108b195b484e21223d27e7a50ba6b" gracePeriod=30
Sep 10 19:37:34.517173 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.517119    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"]
Sep 10 19:37:34.517415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.517388    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w" podUID="3846470e-e119-4e04-a3e4-93d372f5ca56" containerName="payload-processing" containerID="cri-o://fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2" gracePeriod=30
Sep 10 19:37:34.620678 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.620642    2573 generic.go:356] "Generic (PLEG): container finished" podID="9b970999-758d-4c05-be9d-ab87cf1855d8" containerID="e31819cf326f40513c417b5ae2ff66cbf2d108b195b484e21223d27e7a50ba6b" exitCode=0
Sep 10 19:37:34.620871 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.620731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq" event={"ID":"9b970999-758d-4c05-be9d-ab87cf1855d8","Type":"ContainerDied","Data":"e31819cf326f40513c417b5ae2ff66cbf2d108b195b484e21223d27e7a50ba6b"}
Sep 10 19:37:34.774230 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.774203    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:37:34.787831 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.787806    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:37:34.796624 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.796581    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b970999-758d-4c05-be9d-ab87cf1855d8-openshift-service-ca\") pod \"9b970999-758d-4c05-be9d-ab87cf1855d8\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") "
Sep 10 19:37:34.796772 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.796641    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-maas-api-tls\") pod \"9b970999-758d-4c05-be9d-ab87cf1855d8\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") "
Sep 10 19:37:34.796772 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.796717    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fms7g\" (UniqueName: \"kubernetes.io/projected/9b970999-758d-4c05-be9d-ab87cf1855d8-kube-api-access-fms7g\") pod \"9b970999-758d-4c05-be9d-ab87cf1855d8\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") "
Sep 10 19:37:34.796902 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.796803    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-metrics-certs\") pod \"9b970999-758d-4c05-be9d-ab87cf1855d8\" (UID: \"9b970999-758d-4c05-be9d-ab87cf1855d8\") "
Sep 10 19:37:34.796999 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.796975    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b970999-758d-4c05-be9d-ab87cf1855d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b970999-758d-4c05-be9d-ab87cf1855d8" (UID: "9b970999-758d-4c05-be9d-ab87cf1855d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:37:34.797231 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.797209    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b970999-758d-4c05-be9d-ab87cf1855d8-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:34.799130 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.799096    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b970999-758d-4c05-be9d-ab87cf1855d8" (UID: "9b970999-758d-4c05-be9d-ab87cf1855d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:37:34.799415 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.799395    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b970999-758d-4c05-be9d-ab87cf1855d8-kube-api-access-fms7g" (OuterVolumeSpecName: "kube-api-access-fms7g") pod "9b970999-758d-4c05-be9d-ab87cf1855d8" (UID: "9b970999-758d-4c05-be9d-ab87cf1855d8"). InnerVolumeSpecName "kube-api-access-fms7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:37:34.799513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.799493    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b970999-758d-4c05-be9d-ab87cf1855d8" (UID: "9b970999-758d-4c05-be9d-ab87cf1855d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:37:34.898493 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.898461    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbmfb\" (UniqueName: \"kubernetes.io/projected/3846470e-e119-4e04-a3e4-93d372f5ca56-kube-api-access-kbmfb\") pod \"3846470e-e119-4e04-a3e4-93d372f5ca56\" (UID: \"3846470e-e119-4e04-a3e4-93d372f5ca56\") "
Sep 10 19:37:34.898697 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.898518    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3846470e-e119-4e04-a3e4-93d372f5ca56-plugins-config-volume\") pod \"3846470e-e119-4e04-a3e4-93d372f5ca56\" (UID: \"3846470e-e119-4e04-a3e4-93d372f5ca56\") "
Sep 10 19:37:34.898800 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.898785    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:34.898859 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.898801    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fms7g\" (UniqueName: \"kubernetes.io/projected/9b970999-758d-4c05-be9d-ab87cf1855d8-kube-api-access-fms7g\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:34.898859 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.898812    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b970999-758d-4c05-be9d-ab87cf1855d8-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:34.898939 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.898896    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3846470e-e119-4e04-a3e4-93d372f5ca56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3846470e-e119-4e04-a3e4-93d372f5ca56" (UID: "3846470e-e119-4e04-a3e4-93d372f5ca56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:37:34.900749 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.900727    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3846470e-e119-4e04-a3e4-93d372f5ca56-kube-api-access-kbmfb" (OuterVolumeSpecName: "kube-api-access-kbmfb") pod "3846470e-e119-4e04-a3e4-93d372f5ca56" (UID: "3846470e-e119-4e04-a3e4-93d372f5ca56"). InnerVolumeSpecName "kube-api-access-kbmfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:37:34.999632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.999593    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmfb\" (UniqueName: \"kubernetes.io/projected/3846470e-e119-4e04-a3e4-93d372f5ca56-kube-api-access-kbmfb\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:34.999632 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:34.999624    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3846470e-e119-4e04-a3e4-93d372f5ca56-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:37:35.627791 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.627752    2573 generic.go:356] "Generic (PLEG): container finished" podID="3846470e-e119-4e04-a3e4-93d372f5ca56" containerID="fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2" exitCode=0
Sep 10 19:37:35.628322 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.627807    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"
Sep 10 19:37:35.628322 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.627850    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w" event={"ID":"3846470e-e119-4e04-a3e4-93d372f5ca56","Type":"ContainerDied","Data":"fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2"}
Sep 10 19:37:35.628322 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.627918    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w" event={"ID":"3846470e-e119-4e04-a3e4-93d372f5ca56","Type":"ContainerDied","Data":"54b1702794796d408f48d9e3a8d8602f9df142712aeff71afd81bfc842db2ea0"}
Sep 10 19:37:35.628322 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.627955    2573 scope.go:117] "RemoveContainer" containerID="fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2"
Sep 10 19:37:35.629836 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.629809    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"
Sep 10 19:37:35.629921 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.629837    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq" event={"ID":"9b970999-758d-4c05-be9d-ab87cf1855d8","Type":"ContainerDied","Data":"02008cb28a36653be66a789a79b24046ca6178b5c75208e3520d71602df51964"}
Sep 10 19:37:35.637552 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.637533    2573 scope.go:117] "RemoveContainer" containerID="fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2"
Sep 10 19:37:35.637889 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:37:35.637864    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2\": container with ID starting with fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2 not found: ID does not exist" containerID="fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2"
Sep 10 19:37:35.638001 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.637893    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2"} err="failed to get container status \"fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2\": rpc error: code = NotFound desc = could not find container \"fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2\": container with ID starting with fac481e750630925dce57907fc89d45ceb751d22be5abf49e55242d07732fce2 not found: ID does not exist"
Sep 10 19:37:35.638001 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.637909    2573 scope.go:117] "RemoveContainer" containerID="e31819cf326f40513c417b5ae2ff66cbf2d108b195b484e21223d27e7a50ba6b"
Sep 10 19:37:35.648513 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.648491    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"]
Sep 10 19:37:35.654132 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.654107    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c59b36-847c66657f-zqpqq"]
Sep 10 19:37:35.664903 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.664849    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"]
Sep 10 19:37:35.667003 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:35.666983    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c59b36-675d5b6b5-nwd5w"]
Sep 10 19:37:36.834210 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:36.834176    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3846470e-e119-4e04-a3e4-93d372f5ca56" path="/var/lib/kubelet/pods/3846470e-e119-4e04-a3e4-93d372f5ca56/volumes"
Sep 10 19:37:36.834600 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:37:36.834559    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b970999-758d-4c05-be9d-ab87cf1855d8" path="/var/lib/kubelet/pods/9b970999-758d-4c05-be9d-ab87cf1855d8/volumes"
Sep 10 19:38:34.325197 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.325144    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"]
Sep 10 19:38:34.325802 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.325447    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4" podUID="99fcb4e3-f28c-420c-98e4-07484485fc51" containerName="payload-processing" containerID="cri-o://f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569" gracePeriod=30
Sep 10 19:38:34.583200 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.583120    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:38:34.687399 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.687357    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv2kc\" (UniqueName: \"kubernetes.io/projected/99fcb4e3-f28c-420c-98e4-07484485fc51-kube-api-access-cv2kc\") pod \"99fcb4e3-f28c-420c-98e4-07484485fc51\" (UID: \"99fcb4e3-f28c-420c-98e4-07484485fc51\") "
Sep 10 19:38:34.687579 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.687447    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99fcb4e3-f28c-420c-98e4-07484485fc51-plugins-config-volume\") pod \"99fcb4e3-f28c-420c-98e4-07484485fc51\" (UID: \"99fcb4e3-f28c-420c-98e4-07484485fc51\") "
Sep 10 19:38:34.687824 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.687800    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99fcb4e3-f28c-420c-98e4-07484485fc51-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99fcb4e3-f28c-420c-98e4-07484485fc51" (UID: "99fcb4e3-f28c-420c-98e4-07484485fc51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:38:34.689769 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.689748    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fcb4e3-f28c-420c-98e4-07484485fc51-kube-api-access-cv2kc" (OuterVolumeSpecName: "kube-api-access-cv2kc") pod "99fcb4e3-f28c-420c-98e4-07484485fc51" (UID: "99fcb4e3-f28c-420c-98e4-07484485fc51"). InnerVolumeSpecName "kube-api-access-cv2kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:38:34.788277 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.788241    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2kc\" (UniqueName: \"kubernetes.io/projected/99fcb4e3-f28c-420c-98e4-07484485fc51-kube-api-access-cv2kc\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:38:34.788277 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.788269    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99fcb4e3-f28c-420c-98e4-07484485fc51-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:38:34.942211 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.942181    2573 generic.go:356] "Generic (PLEG): container finished" podID="99fcb4e3-f28c-420c-98e4-07484485fc51" containerID="f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569" exitCode=0
Sep 10 19:38:34.942405 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.942238    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4" event={"ID":"99fcb4e3-f28c-420c-98e4-07484485fc51","Type":"ContainerDied","Data":"f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569"}
Sep 10 19:38:34.942405 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.942281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4" event={"ID":"99fcb4e3-f28c-420c-98e4-07484485fc51","Type":"ContainerDied","Data":"25c8665bb238a37bcd8559bc0dfb22368b5d5b100cd2a2257580b79cb1bc86c2"}
Sep 10 19:38:34.942405 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.942288    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"
Sep 10 19:38:34.942405 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.942301    2573 scope.go:117] "RemoveContainer" containerID="f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569"
Sep 10 19:38:34.951345 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.951325    2573 scope.go:117] "RemoveContainer" containerID="f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569"
Sep 10 19:38:34.951637 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:38:34.951617    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569\": container with ID starting with f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569 not found: ID does not exist" containerID="f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569"
Sep 10 19:38:34.951680 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.951647    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569"} err="failed to get container status \"f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569\": rpc error: code = NotFound desc = could not find container \"f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569\": container with ID starting with f5db59760e1a35ba8060247036001c76c6c5738b4b8f9d75201e868bcbc25569 not found: ID does not exist"
Sep 10 19:38:34.963950 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.963913    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"]
Sep 10 19:38:34.965636 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:34.965616    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-90ceb3-7f4bc6cf4c-8hpj4"]
Sep 10 19:38:36.834607 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:36.834565    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99fcb4e3-f28c-420c-98e4-07484485fc51" path="/var/lib/kubelet/pods/99fcb4e3-f28c-420c-98e4-07484485fc51/volumes"
Sep 10 19:38:52.328007 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:52.327966    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"]
Sep 10 19:38:52.328495 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:52.328237    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4" podUID="88d42752-7e3e-4c22-b53c-be9928ec9b95" containerName="payload-pre-processing" containerID="cri-o://f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a" gracePeriod=30
Sep 10 19:38:52.579169 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:52.579091    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:38:52.657714 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:52.657682    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d42752-7e3e-4c22-b53c-be9928ec9b95-plugins-config-volume\") pod \"88d42752-7e3e-4c22-b53c-be9928ec9b95\" (UID: \"88d42752-7e3e-4c22-b53c-be9928ec9b95\") "
Sep 10 19:38:52.657902 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:52.657724    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/88d42752-7e3e-4c22-b53c-be9928ec9b95-kube-api-access-h74j4\") pod \"88d42752-7e3e-4c22-b53c-be9928ec9b95\" (UID: \"88d42752-7e3e-4c22-b53c-be9928ec9b95\") "
Sep 10 19:38:52.658090 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:52.658066    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d42752-7e3e-4c22-b53c-be9928ec9b95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88d42752-7e3e-4c22-b53c-be9928ec9b95" (UID: "88d42752-7e3e-4c22-b53c-be9928ec9b95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:38:52.660065 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:52.660045    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d42752-7e3e-4c22-b53c-be9928ec9b95-kube-api-access-h74j4" (OuterVolumeSpecName: "kube-api-access-h74j4") pod "88d42752-7e3e-4c22-b53c-be9928ec9b95" (UID: "88d42752-7e3e-4c22-b53c-be9928ec9b95"). InnerVolumeSpecName "kube-api-access-h74j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:38:52.759344 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:52.759300    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d42752-7e3e-4c22-b53c-be9928ec9b95-plugins-config-volume\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:38:52.759344 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:52.759334    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h74j4\" (UniqueName: \"kubernetes.io/projected/88d42752-7e3e-4c22-b53c-be9928ec9b95-kube-api-access-h74j4\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:38:53.039400 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:53.039367    2573 generic.go:356] "Generic (PLEG): container finished" podID="88d42752-7e3e-4c22-b53c-be9928ec9b95" containerID="f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a" exitCode=0
Sep 10 19:38:53.039400 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:53.039403    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4" event={"ID":"88d42752-7e3e-4c22-b53c-be9928ec9b95","Type":"ContainerDied","Data":"f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a"}
Sep 10 19:38:53.039636 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:53.039406    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"
Sep 10 19:38:53.039636 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:53.039453    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4" event={"ID":"88d42752-7e3e-4c22-b53c-be9928ec9b95","Type":"ContainerDied","Data":"6eb88406491faa97077b0645f24ad98df0e519dad259ccc5c46e65aa1e275004"}
Sep 10 19:38:53.039636 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:53.039476    2573 scope.go:117] "RemoveContainer" containerID="f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a"
Sep 10 19:38:53.048320 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:53.048301    2573 scope.go:117] "RemoveContainer" containerID="f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a"
Sep 10 19:38:53.048622 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:38:53.048604    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a\": container with ID starting with f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a not found: ID does not exist" containerID="f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a"
Sep 10 19:38:53.048681 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:53.048632    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a"} err="failed to get container status \"f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a\": rpc error: code = NotFound desc = could not find container \"f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a\": container with ID starting with f6398f4432737d94034555c44d28c24b910e7feb639dc0a0a6b15cc87c06263a not found: ID does not exist"
Sep 10 19:38:53.058778 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:53.058748    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"]
Sep 10 19:38:53.063672 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:53.063650    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b64b1c-b5dbc75f8-zdwh4"]
Sep 10 19:38:54.840673 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:38:54.840633    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d42752-7e3e-4c22-b53c-be9928ec9b95" path="/var/lib/kubelet/pods/88d42752-7e3e-4c22-b53c-be9928ec9b95/volumes"
Sep 10 19:39:10.661662 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:10.661623    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"]
Sep 10 19:39:10.662270 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:10.661919    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5" podUID="ba88c0a7-40fa-4ded-b076-89fce928063b" containerName="maas-api" containerID="cri-o://662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d" gracePeriod=30
Sep 10 19:39:10.913890 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:10.913820    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:39:11.033731 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.033695    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba88c0a7-40fa-4ded-b076-89fce928063b-openshift-service-ca\") pod \"ba88c0a7-40fa-4ded-b076-89fce928063b\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") "
Sep 10 19:39:11.033911 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.033761    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-maas-api-tls\") pod \"ba88c0a7-40fa-4ded-b076-89fce928063b\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") "
Sep 10 19:39:11.033911 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.033837    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpfnm\" (UniqueName: \"kubernetes.io/projected/ba88c0a7-40fa-4ded-b076-89fce928063b-kube-api-access-bpfnm\") pod \"ba88c0a7-40fa-4ded-b076-89fce928063b\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") "
Sep 10 19:39:11.033911 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.033866    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-metrics-certs\") pod \"ba88c0a7-40fa-4ded-b076-89fce928063b\" (UID: \"ba88c0a7-40fa-4ded-b076-89fce928063b\") "
Sep 10 19:39:11.034127 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.034096    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba88c0a7-40fa-4ded-b076-89fce928063b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba88c0a7-40fa-4ded-b076-89fce928063b" (UID: "ba88c0a7-40fa-4ded-b076-89fce928063b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:39:11.036028 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.036002    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba88c0a7-40fa-4ded-b076-89fce928063b-kube-api-access-bpfnm" (OuterVolumeSpecName: "kube-api-access-bpfnm") pod "ba88c0a7-40fa-4ded-b076-89fce928063b" (UID: "ba88c0a7-40fa-4ded-b076-89fce928063b"). InnerVolumeSpecName "kube-api-access-bpfnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:39:11.036173 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.036104    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ba88c0a7-40fa-4ded-b076-89fce928063b" (UID: "ba88c0a7-40fa-4ded-b076-89fce928063b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:39:11.036173 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.036129    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba88c0a7-40fa-4ded-b076-89fce928063b" (UID: "ba88c0a7-40fa-4ded-b076-89fce928063b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:39:11.128502 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.128467    2573 generic.go:356] "Generic (PLEG): container finished" podID="ba88c0a7-40fa-4ded-b076-89fce928063b" containerID="662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d" exitCode=0
Sep 10 19:39:11.128664 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.128506    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5" event={"ID":"ba88c0a7-40fa-4ded-b076-89fce928063b","Type":"ContainerDied","Data":"662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d"}
Sep 10 19:39:11.128664 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.128511    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"
Sep 10 19:39:11.128664 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.128554    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5" event={"ID":"ba88c0a7-40fa-4ded-b076-89fce928063b","Type":"ContainerDied","Data":"bd259ad319c49dbbbb4df4e5099fe57259d96143819c3a4cce9337cb58a9be0a"}
Sep 10 19:39:11.128664 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.128577    2573 scope.go:117] "RemoveContainer" containerID="662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d"
Sep 10 19:39:11.134826 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.134803    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba88c0a7-40fa-4ded-b076-89fce928063b-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:39:11.134935 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.134828    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:39:11.134935 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.134842    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpfnm\" (UniqueName: \"kubernetes.io/projected/ba88c0a7-40fa-4ded-b076-89fce928063b-kube-api-access-bpfnm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:39:11.134935 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.134854    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba88c0a7-40fa-4ded-b076-89fce928063b-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:39:11.138638 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.138620    2573 scope.go:117] "RemoveContainer" containerID="662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d"
Sep 10 19:39:11.138911 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:39:11.138889    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d\": container with ID starting with 662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d not found: ID does not exist" containerID="662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d"
Sep 10 19:39:11.138957 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.138920    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d"} err="failed to get container status \"662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d\": rpc error: code = NotFound desc = could not find container \"662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d\": container with ID starting with 662ebc22670472a74ffd72e88f2a6e775aa87e405f5ef153d3aab41dce13719d not found: ID does not exist"
Sep 10 19:39:11.152865 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.152842    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"]
Sep 10 19:39:11.157053 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:11.157034    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f15c09-5ff879f7c7-8scm5"]
Sep 10 19:39:12.834556 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:39:12.834523    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba88c0a7-40fa-4ded-b076-89fce928063b" path="/var/lib/kubelet/pods/ba88c0a7-40fa-4ded-b076-89fce928063b/volumes"
Sep 10 19:43:56.926196 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926095    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"]
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926583    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99fcb4e3-f28c-420c-98e4-07484485fc51" containerName="payload-processing"
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926600    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="99fcb4e3-f28c-420c-98e4-07484485fc51" containerName="payload-processing"
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926621    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3846470e-e119-4e04-a3e4-93d372f5ca56" containerName="payload-processing"
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926626    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3846470e-e119-4e04-a3e4-93d372f5ca56" containerName="payload-processing"
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926639    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba88c0a7-40fa-4ded-b076-89fce928063b" containerName="maas-api"
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926645    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba88c0a7-40fa-4ded-b076-89fce928063b" containerName="maas-api"
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926652    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d42752-7e3e-4c22-b53c-be9928ec9b95" containerName="payload-pre-processing"
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926658    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d42752-7e3e-4c22-b53c-be9928ec9b95" containerName="payload-pre-processing"
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926664    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2a949a4-2aa0-466e-b340-02b3e5fcb408" containerName="payload-pre-processing"
Sep 10 19:43:56.926669 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926670    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2a949a4-2aa0-466e-b340-02b3e5fcb408" containerName="payload-pre-processing"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926680    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0683e501-1727-482e-a54b-750606d43e5f" containerName="maas-api"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926686    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0683e501-1727-482e-a54b-750606d43e5f" containerName="maas-api"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926693    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b970999-758d-4c05-be9d-ab87cf1855d8" containerName="maas-api"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926698    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b970999-758d-4c05-be9d-ab87cf1855d8" containerName="maas-api"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926752    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba88c0a7-40fa-4ded-b076-89fce928063b" containerName="maas-api"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926761    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2a949a4-2aa0-466e-b340-02b3e5fcb408" containerName="payload-pre-processing"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926773    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b970999-758d-4c05-be9d-ab87cf1855d8" containerName="maas-api"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926781    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d42752-7e3e-4c22-b53c-be9928ec9b95" containerName="payload-pre-processing"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926791    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="99fcb4e3-f28c-420c-98e4-07484485fc51" containerName="payload-processing"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926799    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3846470e-e119-4e04-a3e4-93d372f5ca56" containerName="payload-processing"
Sep 10 19:43:56.927004 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.926807    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0683e501-1727-482e-a54b-750606d43e5f" containerName="maas-api"
Sep 10 19:43:56.929969 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.929951    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:56.933441 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.933420    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d6143394-kserve-self-signed-certs\""
Sep 10 19:43:56.941383 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:56.941353    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"]
Sep 10 19:43:57.122076 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.122038    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-home\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.122295 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.122094    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-tmp-dir\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.122295 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.122231    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dswq\" (UniqueName: \"kubernetes.io/projected/34166688-2b64-42bf-b1fa-5edd46c88eb0-kube-api-access-9dswq\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.122295 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.122282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-dshm\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.122436 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.122335    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-model-cache\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.122436 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.122365    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166688-2b64-42bf-b1fa-5edd46c88eb0-tls-certs\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.223724 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.223626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-tmp-dir\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.223724 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.223692    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dswq\" (UniqueName: \"kubernetes.io/projected/34166688-2b64-42bf-b1fa-5edd46c88eb0-kube-api-access-9dswq\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.223974 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.223727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-dshm\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.223974 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.223755    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-model-cache\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.223974 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.223785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166688-2b64-42bf-b1fa-5edd46c88eb0-tls-certs\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.223974 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.223893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-home\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.224226 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.224079    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-tmp-dir\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.224226 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.224119    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-model-cache\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.224339 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.224293    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-home\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.226231 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.226196    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-dshm\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.226370 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.226351    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166688-2b64-42bf-b1fa-5edd46c88eb0-tls-certs\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.237237 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.237207    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dswq\" (UniqueName: \"kubernetes.io/projected/34166688-2b64-42bf-b1fa-5edd46c88eb0-kube-api-access-9dswq\") pod \"e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") " pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.240039 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.240014    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:43:57.521875 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.521838    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"]
Sep 10 19:43:57.525536 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:43:57.525503    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34166688_2b64_42bf_b1fa_5edd46c88eb0.slice/crio-2ecb0c3fe4ca7b30ad420e478240b129123ec4bc0965a503438e889a0c4becd4 WatchSource:0}: Error finding container 2ecb0c3fe4ca7b30ad420e478240b129123ec4bc0965a503438e889a0c4becd4: Status 404 returned error can't find the container with id 2ecb0c3fe4ca7b30ad420e478240b129123ec4bc0965a503438e889a0c4becd4
Sep 10 19:43:57.540699 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:57.540655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8" event={"ID":"34166688-2b64-42bf-b1fa-5edd46c88eb0","Type":"ContainerStarted","Data":"2ecb0c3fe4ca7b30ad420e478240b129123ec4bc0965a503438e889a0c4becd4"}
Sep 10 19:43:58.546610 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:58.546569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8" event={"ID":"34166688-2b64-42bf-b1fa-5edd46c88eb0","Type":"ContainerStarted","Data":"2cf9b69e5b53e08315815589ddc25934216f0b1dd6b03fd873900e202c9d8d5b"}
Sep 10 19:43:58.570590 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:43:58.570539    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8" podStartSLOduration=2.570519199 podStartE2EDuration="2.570519199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:43:58.568958466 +0000 UTC m=+1974.228755684" watchObservedRunningTime="2026-09-10 19:43:58.570519199 +0000 UTC m=+1974.230316392"
Sep 10 19:44:07.241104 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:07.241062    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:44:07.241535 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:07.241132    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:44:07.253727 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:07.253693    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:44:07.602183 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:07.602133    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:44:18.468828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:18.468792    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"]
Sep 10 19:44:33.489785 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.489721    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8" podUID="34166688-2b64-42bf-b1fa-5edd46c88eb0" containerName="main" containerID="cri-o://2cf9b69e5b53e08315815589ddc25934216f0b1dd6b03fd873900e202c9d8d5b" gracePeriod=45
Sep 10 19:44:33.723065 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.723037    2573 generic.go:356] "Generic (PLEG): container finished" podID="34166688-2b64-42bf-b1fa-5edd46c88eb0" containerID="2cf9b69e5b53e08315815589ddc25934216f0b1dd6b03fd873900e202c9d8d5b" exitCode=0
Sep 10 19:44:33.723236 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.723112    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8" event={"ID":"34166688-2b64-42bf-b1fa-5edd46c88eb0","Type":"ContainerDied","Data":"2cf9b69e5b53e08315815589ddc25934216f0b1dd6b03fd873900e202c9d8d5b"}
Sep 10 19:44:33.743516 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.743444    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:44:33.880286 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880245    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-home\") pod \"34166688-2b64-42bf-b1fa-5edd46c88eb0\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") "
Sep 10 19:44:33.880286 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880283    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-dshm\") pod \"34166688-2b64-42bf-b1fa-5edd46c88eb0\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") "
Sep 10 19:44:33.880551 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880321    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-tmp-dir\") pod \"34166688-2b64-42bf-b1fa-5edd46c88eb0\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") "
Sep 10 19:44:33.880551 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880354    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-model-cache\") pod \"34166688-2b64-42bf-b1fa-5edd46c88eb0\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") "
Sep 10 19:44:33.880551 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880425    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dswq\" (UniqueName: \"kubernetes.io/projected/34166688-2b64-42bf-b1fa-5edd46c88eb0-kube-api-access-9dswq\") pod \"34166688-2b64-42bf-b1fa-5edd46c88eb0\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") "
Sep 10 19:44:33.880551 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880463    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166688-2b64-42bf-b1fa-5edd46c88eb0-tls-certs\") pod \"34166688-2b64-42bf-b1fa-5edd46c88eb0\" (UID: \"34166688-2b64-42bf-b1fa-5edd46c88eb0\") "
Sep 10 19:44:33.880739 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880552    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-home" (OuterVolumeSpecName: "home") pod "34166688-2b64-42bf-b1fa-5edd46c88eb0" (UID: "34166688-2b64-42bf-b1fa-5edd46c88eb0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:44:33.880739 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880640    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34166688-2b64-42bf-b1fa-5edd46c88eb0" (UID: "34166688-2b64-42bf-b1fa-5edd46c88eb0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:44:33.880739 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880653    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-model-cache" (OuterVolumeSpecName: "model-cache") pod "34166688-2b64-42bf-b1fa-5edd46c88eb0" (UID: "34166688-2b64-42bf-b1fa-5edd46c88eb0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:44:33.880843 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880752    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-home\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:33.880843 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880766    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-tmp-dir\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:33.880843 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.880775    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-model-cache\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:33.882572 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.882543    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-dshm" (OuterVolumeSpecName: "dshm") pod "34166688-2b64-42bf-b1fa-5edd46c88eb0" (UID: "34166688-2b64-42bf-b1fa-5edd46c88eb0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:44:33.882733 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.882717    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34166688-2b64-42bf-b1fa-5edd46c88eb0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34166688-2b64-42bf-b1fa-5edd46c88eb0" (UID: "34166688-2b64-42bf-b1fa-5edd46c88eb0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:44:33.882797 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.882776    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34166688-2b64-42bf-b1fa-5edd46c88eb0-kube-api-access-9dswq" (OuterVolumeSpecName: "kube-api-access-9dswq") pod "34166688-2b64-42bf-b1fa-5edd46c88eb0" (UID: "34166688-2b64-42bf-b1fa-5edd46c88eb0"). InnerVolumeSpecName "kube-api-access-9dswq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:44:33.981468 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.981424    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34166688-2b64-42bf-b1fa-5edd46c88eb0-dshm\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:33.981678 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.981463    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dswq\" (UniqueName: \"kubernetes.io/projected/34166688-2b64-42bf-b1fa-5edd46c88eb0-kube-api-access-9dswq\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:33.981678 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:33.981496    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34166688-2b64-42bf-b1fa-5edd46c88eb0-tls-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:34.730014 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:34.729985    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"
Sep 10 19:44:34.730464 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:34.730027    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8" event={"ID":"34166688-2b64-42bf-b1fa-5edd46c88eb0","Type":"ContainerDied","Data":"2ecb0c3fe4ca7b30ad420e478240b129123ec4bc0965a503438e889a0c4becd4"}
Sep 10 19:44:34.730464 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:34.730086    2573 scope.go:117] "RemoveContainer" containerID="2cf9b69e5b53e08315815589ddc25934216f0b1dd6b03fd873900e202c9d8d5b"
Sep 10 19:44:34.753429 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:34.753396    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"]
Sep 10 19:44:34.757555 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:34.757516    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d6143394-kserve-64bb57fb7f-jqbn8"]
Sep 10 19:44:34.834827 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:34.834793    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34166688-2b64-42bf-b1fa-5edd46c88eb0" path="/var/lib/kubelet/pods/34166688-2b64-42bf-b1fa-5edd46c88eb0/volumes"
Sep 10 19:44:37.517197 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.517136    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"]
Sep 10 19:44:37.517620 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.517570    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34166688-2b64-42bf-b1fa-5edd46c88eb0" containerName="main"
Sep 10 19:44:37.517620 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.517583    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="34166688-2b64-42bf-b1fa-5edd46c88eb0" containerName="main"
Sep 10 19:44:37.517696 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.517643    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="34166688-2b64-42bf-b1fa-5edd46c88eb0" containerName="main"
Sep 10 19:44:37.520605 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.520577    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.523928 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.523903    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 19:44:37.524950 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.524927    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lw7bv\""
Sep 10 19:44:37.525085 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.524978    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 19:44:37.530980 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.530951    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"]
Sep 10 19:44:37.618380 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.618339    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da81ed-bddf-40d9-aea6-584a0aa9af15-openshift-service-ca\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.618601 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.618397    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-metrics-certs\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.618601 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.618549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-maas-api-tls\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.618601 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.618590    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wsjp\" (UniqueName: \"kubernetes.io/projected/b3da81ed-bddf-40d9-aea6-584a0aa9af15-kube-api-access-7wsjp\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.719224 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.719181    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-metrics-certs\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.719429 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.719263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-maas-api-tls\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.719429 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.719289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wsjp\" (UniqueName: \"kubernetes.io/projected/b3da81ed-bddf-40d9-aea6-584a0aa9af15-kube-api-access-7wsjp\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.719429 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.719329    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da81ed-bddf-40d9-aea6-584a0aa9af15-openshift-service-ca\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.719917 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.719897    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da81ed-bddf-40d9-aea6-584a0aa9af15-openshift-service-ca\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.721728 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.721703    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-maas-api-tls\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.721835 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.721815    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-metrics-certs\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.727308 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.727284    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wsjp\" (UniqueName: \"kubernetes.io/projected/b3da81ed-bddf-40d9-aea6-584a0aa9af15-kube-api-access-7wsjp\") pod \"maas-api-ff9f6d487-zn7z2\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:37.832851 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:37.832811    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:38.108803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:38.108772    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"]
Sep 10 19:44:38.111074 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:44:38.111044    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3da81ed_bddf_40d9_aea6_584a0aa9af15.slice/crio-063c35f183ab875507f476afcc166965e6c9bd84ecfa53b40944a220752b9fa8 WatchSource:0}: Error finding container 063c35f183ab875507f476afcc166965e6c9bd84ecfa53b40944a220752b9fa8: Status 404 returned error can't find the container with id 063c35f183ab875507f476afcc166965e6c9bd84ecfa53b40944a220752b9fa8
Sep 10 19:44:38.112481 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:38.112459    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:44:38.754732 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:38.754697    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2" event={"ID":"b3da81ed-bddf-40d9-aea6-584a0aa9af15","Type":"ContainerStarted","Data":"5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071"}
Sep 10 19:44:38.755311 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:38.754759    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2" event={"ID":"b3da81ed-bddf-40d9-aea6-584a0aa9af15","Type":"ContainerStarted","Data":"063c35f183ab875507f476afcc166965e6c9bd84ecfa53b40944a220752b9fa8"}
Sep 10 19:44:38.755311 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:38.754788    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:38.781186 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:38.781113    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2" podStartSLOduration=1.350749132 podStartE2EDuration="1.781095605s" totalImagesPullingTime="430.346473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:44:38.778604483 +0000 UTC m=+2014.438401677" watchObservedRunningTime="2026-09-10 19:44:38.781095605 +0000 UTC m=+2014.440892857"
Sep 10 19:44:44.764752 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:44.764721    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:45.375419 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.375382    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"]
Sep 10 19:44:45.379290 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.379270    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.391687 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.391657    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"]
Sep 10 19:44:45.492223 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.492140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2edc0405-d48a-431a-8db7-7f4df1c71283-maas-api-tls\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.492223 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.492221    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2edc0405-d48a-431a-8db7-7f4df1c71283-metrics-certs\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.492447 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.492269    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmrn\" (UniqueName: \"kubernetes.io/projected/2edc0405-d48a-431a-8db7-7f4df1c71283-kube-api-access-wjmrn\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.492447 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.492344    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2edc0405-d48a-431a-8db7-7f4df1c71283-openshift-service-ca\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.593900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.593859    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2edc0405-d48a-431a-8db7-7f4df1c71283-maas-api-tls\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.593900 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.593906    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2edc0405-d48a-431a-8db7-7f4df1c71283-metrics-certs\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.594214 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.593957    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmrn\" (UniqueName: \"kubernetes.io/projected/2edc0405-d48a-431a-8db7-7f4df1c71283-kube-api-access-wjmrn\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.594214 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.593989    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2edc0405-d48a-431a-8db7-7f4df1c71283-openshift-service-ca\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.594655 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.594634    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2edc0405-d48a-431a-8db7-7f4df1c71283-openshift-service-ca\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.596522 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.596499    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2edc0405-d48a-431a-8db7-7f4df1c71283-maas-api-tls\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.596623 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.596572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2edc0405-d48a-431a-8db7-7f4df1c71283-metrics-certs\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.602555 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.602528    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmrn\" (UniqueName: \"kubernetes.io/projected/2edc0405-d48a-431a-8db7-7f4df1c71283-kube-api-access-wjmrn\") pod \"maas-api-777cfbf6dc-v6bqf\" (UID: \"2edc0405-d48a-431a-8db7-7f4df1c71283\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.692862 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.692766    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:45.972329 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:45.972302    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"]
Sep 10 19:44:45.974579 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:44:45.974550    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edc0405_d48a_431a_8db7_7f4df1c71283.slice/crio-cc25185a594a31feb3e367d3e666382fc3566554fd40c7e000d623006a0cb3e1 WatchSource:0}: Error finding container cc25185a594a31feb3e367d3e666382fc3566554fd40c7e000d623006a0cb3e1: Status 404 returned error can't find the container with id cc25185a594a31feb3e367d3e666382fc3566554fd40c7e000d623006a0cb3e1
Sep 10 19:44:46.799094 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:46.799060    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf" event={"ID":"2edc0405-d48a-431a-8db7-7f4df1c71283","Type":"ContainerStarted","Data":"c86120591dfd6af01cbe26e72d71da3a67058d9baf8803bcef16956ad963775a"}
Sep 10 19:44:46.799312 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:46.799109    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf" event={"ID":"2edc0405-d48a-431a-8db7-7f4df1c71283","Type":"ContainerStarted","Data":"cc25185a594a31feb3e367d3e666382fc3566554fd40c7e000d623006a0cb3e1"}
Sep 10 19:44:46.799312 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:46.799215    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:46.821599 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:46.821542    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf" podStartSLOduration=1.4949093150000001 podStartE2EDuration="1.821523411s" totalImagesPullingTime="326.614096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:44:46.819806314 +0000 UTC m=+2022.479603507" watchObservedRunningTime="2026-09-10 19:44:46.821523411 +0000 UTC m=+2022.481320606"
Sep 10 19:44:52.811506 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:52.811462    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-v6bqf"
Sep 10 19:44:52.892694 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:52.892656    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"]
Sep 10 19:44:52.892941 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:52.892899    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2" podUID="b3da81ed-bddf-40d9-aea6-584a0aa9af15" containerName="maas-api" containerID="cri-o://5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071" gracePeriod=30
Sep 10 19:44:54.144540 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.144507    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:54.282347 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.282319    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wsjp\" (UniqueName: \"kubernetes.io/projected/b3da81ed-bddf-40d9-aea6-584a0aa9af15-kube-api-access-7wsjp\") pod \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") "
Sep 10 19:44:54.282527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.282357    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-metrics-certs\") pod \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") "
Sep 10 19:44:54.282527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.282433    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da81ed-bddf-40d9-aea6-584a0aa9af15-openshift-service-ca\") pod \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") "
Sep 10 19:44:54.282527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.282491    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-maas-api-tls\") pod \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\" (UID: \"b3da81ed-bddf-40d9-aea6-584a0aa9af15\") "
Sep 10 19:44:54.282868 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.282840    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3da81ed-bddf-40d9-aea6-584a0aa9af15-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3da81ed-bddf-40d9-aea6-584a0aa9af15" (UID: "b3da81ed-bddf-40d9-aea6-584a0aa9af15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:44:54.284784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.284748    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b3da81ed-bddf-40d9-aea6-584a0aa9af15" (UID: "b3da81ed-bddf-40d9-aea6-584a0aa9af15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:44:54.284784 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.284760    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3da81ed-bddf-40d9-aea6-584a0aa9af15-kube-api-access-7wsjp" (OuterVolumeSpecName: "kube-api-access-7wsjp") pod "b3da81ed-bddf-40d9-aea6-584a0aa9af15" (UID: "b3da81ed-bddf-40d9-aea6-584a0aa9af15"). InnerVolumeSpecName "kube-api-access-7wsjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:44:54.284916 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.284760    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3da81ed-bddf-40d9-aea6-584a0aa9af15" (UID: "b3da81ed-bddf-40d9-aea6-584a0aa9af15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:44:54.384119 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.384080    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3da81ed-bddf-40d9-aea6-584a0aa9af15-openshift-service-ca\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:54.384119 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.384112    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-maas-api-tls\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:54.384119 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.384121    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wsjp\" (UniqueName: \"kubernetes.io/projected/b3da81ed-bddf-40d9-aea6-584a0aa9af15-kube-api-access-7wsjp\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:54.384119 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.384129    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3da81ed-bddf-40d9-aea6-584a0aa9af15-metrics-certs\") on node \"ip-10-0-141-45.ec2.internal\" DevicePath \"\""
Sep 10 19:44:54.840267 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.840232    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3da81ed-bddf-40d9-aea6-584a0aa9af15" containerID="5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071" exitCode=0
Sep 10 19:44:54.840456 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.840276    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2" event={"ID":"b3da81ed-bddf-40d9-aea6-584a0aa9af15","Type":"ContainerDied","Data":"5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071"}
Sep 10 19:44:54.840456 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.840315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2" event={"ID":"b3da81ed-bddf-40d9-aea6-584a0aa9af15","Type":"ContainerDied","Data":"063c35f183ab875507f476afcc166965e6c9bd84ecfa53b40944a220752b9fa8"}
Sep 10 19:44:54.840456 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.840336    2573 scope.go:117] "RemoveContainer" containerID="5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071"
Sep 10 19:44:54.840456 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.840278    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"
Sep 10 19:44:54.850751 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.850732    2573 scope.go:117] "RemoveContainer" containerID="5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071"
Sep 10 19:44:54.851135 ip-10-0-141-45 kubenswrapper[2573]: E0910 19:44:54.851114    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071\": container with ID starting with 5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071 not found: ID does not exist" containerID="5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071"
Sep 10 19:44:54.851225 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.851160    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071"} err="failed to get container status \"5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071\": rpc error: code = NotFound desc = could not find container \"5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071\": container with ID starting with 5aefde911f7bf9a8f88e11accbae243937c859f7988cd404f79427aa74606071 not found: ID does not exist"
Sep 10 19:44:54.861927 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.861897    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"]
Sep 10 19:44:54.865759 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:54.865729    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-zn7z2"]
Sep 10 19:44:56.833616 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:44:56.833574    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3da81ed-bddf-40d9-aea6-584a0aa9af15" path="/var/lib/kubelet/pods/b3da81ed-bddf-40d9-aea6-584a0aa9af15/volumes"
Sep 10 19:45:11.309409 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:11.309370    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cd585ddf7-k7xjg_511a22b8-cc23-453d-8b2a-6837afdf8fec/authorino/0.log"
Sep 10 19:45:17.472139 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:17.472107    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-xsg8v_11bd4506-ecf3-4b70-b37f-d39b75348869/copy-manifests/0.log"
Sep 10 19:45:17.481232 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:17.481206    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-xsg8v_11bd4506-ecf3-4b70-b37f-d39b75348869/manager/0.log"
Sep 10 19:45:17.920007 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:17.919977    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-777cfbf6dc-v6bqf_2edc0405-d48a-431a-8db7-7f4df1c71283/maas-api/0.log"
Sep 10 19:45:18.254688 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:18.254598    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-rczrc_3171ecb9-0f9b-4a39-88de-12953b2ae50e/postgres/0.log"
Sep 10 19:45:19.028932 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:19.028897    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cd585ddf7-k7xjg_511a22b8-cc23-453d-8b2a-6837afdf8fec/authorino/0.log"
Sep 10 19:45:19.715663 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:19.715631    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-stz98_17998649-d79e-4c82-91c3-a005e54c004c/manager/0.log"
Sep 10 19:45:20.283917 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:20.283881    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5f7cd94b-cqhc8_075a6585-5c1b-4f46-911d-2f7b26c45cb2/kube-auth-proxy/0.log"
Sep 10 19:45:21.187125 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:21.187096    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-cdxgj_e93b1456-57eb-48fc-b436-db4656447a30/main/0.log"
Sep 10 19:45:21.296741 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:21.296711    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dwfp7_f8c2ebf5-befd-44f2-bac6-e1b9cbddec69/main/0.log"
Sep 10 19:45:30.052562 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:30.052525    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-htz6q_bfc622a4-f58d-42ec-97f6-5271a1a9d3bc/global-pull-secret-syncer/0.log"
Sep 10 19:45:30.270986 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:30.270938    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wn7z8_356a1f2e-3fbb-48a4-b9a1-b8739eef5b8b/konnectivity-agent/0.log"
Sep 10 19:45:30.326505 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:30.326420    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-45.ec2.internal_6cbe649ca88acb6ca8918c435c58b662/haproxy/0.log"
Sep 10 19:45:34.999931 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:34.999894    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cd585ddf7-k7xjg_511a22b8-cc23-453d-8b2a-6837afdf8fec/authorino/0.log"
Sep 10 19:45:35.315354 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:35.315295    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-stz98_17998649-d79e-4c82-91c3-a005e54c004c/manager/0.log"
Sep 10 19:45:36.826453 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:36.826422    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d061bc22-f892-4d24-8d95-857ff39d1ed3/alertmanager/0.log"
Sep 10 19:45:36.852763 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:36.852719    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d061bc22-f892-4d24-8d95-857ff39d1ed3/config-reloader/0.log"
Sep 10 19:45:36.885386 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:36.885354    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d061bc22-f892-4d24-8d95-857ff39d1ed3/kube-rbac-proxy-web/0.log"
Sep 10 19:45:36.912122 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:36.912095    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d061bc22-f892-4d24-8d95-857ff39d1ed3/kube-rbac-proxy/0.log"
Sep 10 19:45:36.938143 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:36.938121    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d061bc22-f892-4d24-8d95-857ff39d1ed3/kube-rbac-proxy-metric/0.log"
Sep 10 19:45:36.964192 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:36.964137    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d061bc22-f892-4d24-8d95-857ff39d1ed3/prom-label-proxy/0.log"
Sep 10 19:45:36.989082 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:36.989057    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d061bc22-f892-4d24-8d95-857ff39d1ed3/init-config-reloader/0.log"
Sep 10 19:45:37.065086 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.065057    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fh2tx_f5330f3a-ae25-4efa-92b6-fb2e6f0054ea/kube-state-metrics/0.log"
Sep 10 19:45:37.089104 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.089076    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fh2tx_f5330f3a-ae25-4efa-92b6-fb2e6f0054ea/kube-rbac-proxy-main/0.log"
Sep 10 19:45:37.115740 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.115712    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-fh2tx_f5330f3a-ae25-4efa-92b6-fb2e6f0054ea/kube-rbac-proxy-self/0.log"
Sep 10 19:45:37.147822 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.147796    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-59597c4497-bl8ch_f7b63c18-d466-49fe-90c3-aa770bc5695a/metrics-server/0.log"
Sep 10 19:45:37.178238 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.178213    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-z8rjz_4f7af8f8-1261-4293-98b4-597e20e24f1b/monitoring-plugin/0.log"
Sep 10 19:45:37.311101 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.311068    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b7tbz_a9f1b138-b565-4acd-a676-568aa20a8321/node-exporter/0.log"
Sep 10 19:45:37.338079 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.338003    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b7tbz_a9f1b138-b565-4acd-a676-568aa20a8321/kube-rbac-proxy/0.log"
Sep 10 19:45:37.367254 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.367230    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b7tbz_a9f1b138-b565-4acd-a676-568aa20a8321/init-textfile/0.log"
Sep 10 19:45:37.514462 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.514436    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2qnp5_8c542347-3526-4891-acd3-a052ded42597/kube-rbac-proxy-main/0.log"
Sep 10 19:45:37.546354 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.546328    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2qnp5_8c542347-3526-4891-acd3-a052ded42597/kube-rbac-proxy-self/0.log"
Sep 10 19:45:37.592889 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.592790    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2qnp5_8c542347-3526-4891-acd3-a052ded42597/openshift-state-metrics/0.log"
Sep 10 19:45:37.656871 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.656840    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65193eb1-624a-4625-ab5b-eb6068942804/prometheus/0.log"
Sep 10 19:45:37.698284 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.698254    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65193eb1-624a-4625-ab5b-eb6068942804/config-reloader/0.log"
Sep 10 19:45:37.730201 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.730173    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65193eb1-624a-4625-ab5b-eb6068942804/thanos-sidecar/0.log"
Sep 10 19:45:37.763055 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.763014    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65193eb1-624a-4625-ab5b-eb6068942804/kube-rbac-proxy-web/0.log"
Sep 10 19:45:37.801930 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.801896    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65193eb1-624a-4625-ab5b-eb6068942804/kube-rbac-proxy/0.log"
Sep 10 19:45:37.828057 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.828023    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65193eb1-624a-4625-ab5b-eb6068942804/kube-rbac-proxy-thanos/0.log"
Sep 10 19:45:37.854681 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.854599    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_65193eb1-624a-4625-ab5b-eb6068942804/init-config-reloader/0.log"
Sep 10 19:45:37.902073 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.902049    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-x2gc4_6b879388-e731-454a-a839-b5ae3572b3c0/prometheus-operator/0.log"
Sep 10 19:45:37.926136 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:37.926111    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-x2gc4_6b879388-e731-454a-a839-b5ae3572b3c0/kube-rbac-proxy/0.log"
Sep 10 19:45:38.012601 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:38.012574    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d56dd9545-wr64z_070da58d-1daa-4ed3-90af-f04b701d9a2c/telemeter-client/0.log"
Sep 10 19:45:38.038963 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:38.038938    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d56dd9545-wr64z_070da58d-1daa-4ed3-90af-f04b701d9a2c/reload/0.log"
Sep 10 19:45:38.072589 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:38.072562    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d56dd9545-wr64z_070da58d-1daa-4ed3-90af-f04b701d9a2c/kube-rbac-proxy/0.log"
Sep 10 19:45:38.109957 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:38.109861    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b5bf859fb-m9ws4_a163d3dd-5faa-488a-bb2a-af839633f2bb/thanos-query/0.log"
Sep 10 19:45:38.137025 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:38.137001    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b5bf859fb-m9ws4_a163d3dd-5faa-488a-bb2a-af839633f2bb/kube-rbac-proxy-web/0.log"
Sep 10 19:45:38.163627 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:38.163601    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b5bf859fb-m9ws4_a163d3dd-5faa-488a-bb2a-af839633f2bb/kube-rbac-proxy/0.log"
Sep 10 19:45:38.193206 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:38.193181    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b5bf859fb-m9ws4_a163d3dd-5faa-488a-bb2a-af839633f2bb/prom-label-proxy/0.log"
Sep 10 19:45:38.223366 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:38.223307    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b5bf859fb-m9ws4_a163d3dd-5faa-488a-bb2a-af839633f2bb/kube-rbac-proxy-rules/0.log"
Sep 10 19:45:38.249858 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:38.249832    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b5bf859fb-m9ws4_a163d3dd-5faa-488a-bb2a-af839633f2bb/kube-rbac-proxy-metrics/0.log"
Sep 10 19:45:40.575872 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.575832    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"]
Sep 10 19:45:40.576324 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.576295    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3da81ed-bddf-40d9-aea6-584a0aa9af15" containerName="maas-api"
Sep 10 19:45:40.576324 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.576319    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3da81ed-bddf-40d9-aea6-584a0aa9af15" containerName="maas-api"
Sep 10 19:45:40.576527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.576458    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3da81ed-bddf-40d9-aea6-584a0aa9af15" containerName="maas-api"
Sep 10 19:45:40.579699 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.579678    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.582499 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.582474    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vr76\"/\"default-dockercfg-j88m6\""
Sep 10 19:45:40.582594 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.582483    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vr76\"/\"openshift-service-ca.crt\""
Sep 10 19:45:40.582594 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.582528    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vr76\"/\"kube-root-ca.crt\""
Sep 10 19:45:40.588706 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.588678    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"]
Sep 10 19:45:40.726549 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.726514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-sys\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.726720 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.726566    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-lib-modules\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.726720 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.726598    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfdh\" (UniqueName: \"kubernetes.io/projected/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-kube-api-access-hcfdh\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.726720 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.726692    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-podres\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.726828 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.726722    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-proc\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.828208 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.828075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-sys\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.828208 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.828132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-lib-modules\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.828208 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.828201    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfdh\" (UniqueName: \"kubernetes.io/projected/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-kube-api-access-hcfdh\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.828495 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.828250    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-sys\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.828495 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.828323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-podres\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.828495 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.828373    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-proc\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.828495 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.828413    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-proc\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.828495 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.828376    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-lib-modules\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.828495 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.828444    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-podres\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.838344 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.838313    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfdh\" (UniqueName: \"kubernetes.io/projected/a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e-kube-api-access-hcfdh\") pod \"perf-node-gather-daemonset-w57fc\" (UID: \"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.868375 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.868330    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54546fdd-2xtb9_52ffe80b-6e8f-4714-a2e4-16827b343745/console/0.log"
Sep 10 19:45:40.890956 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.890920    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:40.913350 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:40.913325    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-6vdrv_41c2f980-3cbf-4fe6-8daf-d90a57fec390/download-server/0.log"
Sep 10 19:45:41.174823 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:41.174797    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"]
Sep 10 19:45:41.177042 ip-10-0-141-45 kubenswrapper[2573]: W0910 19:45:41.177016    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda9b638b5_c8db_4f8a_a5f4_887edfa1ec6e.slice/crio-a40a6a78306d2083cd670b108ca33513e09fdaa9781587407167b546bfdc48cc WatchSource:0}: Error finding container a40a6a78306d2083cd670b108ca33513e09fdaa9781587407167b546bfdc48cc: Status 404 returned error can't find the container with id a40a6a78306d2083cd670b108ca33513e09fdaa9781587407167b546bfdc48cc
Sep 10 19:45:42.070468 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:42.070432    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc" event={"ID":"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e","Type":"ContainerStarted","Data":"4c4ebe933074f629cd68fb0978fe65a3ad4b4a2ce6b855a9bef65a1db0144072"}
Sep 10 19:45:42.070862 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:42.070508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc" event={"ID":"a9b638b5-c8db-4f8a-a5f4-887edfa1ec6e","Type":"ContainerStarted","Data":"a40a6a78306d2083cd670b108ca33513e09fdaa9781587407167b546bfdc48cc"}
Sep 10 19:45:42.070862 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:42.070578    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:42.090519 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:42.090461    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc" podStartSLOduration=2.090442805 podStartE2EDuration="2.090442805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:45:42.089287122 +0000 UTC m=+2077.749084316" watchObservedRunningTime="2026-09-10 19:45:42.090442805 +0000 UTC m=+2077.750239998"
Sep 10 19:45:42.694418 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:42.694386    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8fgm_8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c/dns/0.log"
Sep 10 19:45:42.717514 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:42.717476    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-b8fgm_8f5678b7-5ab3-454f-ae3a-8b1eb6d6d72c/kube-rbac-proxy/0.log"
Sep 10 19:45:42.889049 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:42.889015    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mbsgt_cc8c22ae-6623-4996-a91f-1007228beb5e/dns-node-resolver/0.log"
Sep 10 19:45:43.604735 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:43.604688    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctj27_4d5123b1-49ac-4bcc-99d3-ef530a387f86/node-ca/0.log"
Sep 10 19:45:44.862664 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:44.862635    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5f7cd94b-cqhc8_075a6585-5c1b-4f46-911d-2f7b26c45cb2/kube-auth-proxy/0.log"
Sep 10 19:45:45.934899 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:45.934869    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-97hkg_b1560abb-47cd-4eff-ac93-e3f9f1deb86f/serve-healthcheck-canary/0.log"
Sep 10 19:45:46.508628 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:46.508597    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77nq5_f1110bb8-b19b-49cc-877c-dc28e872558f/kube-rbac-proxy/0.log"
Sep 10 19:45:46.531479 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:46.531447    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77nq5_f1110bb8-b19b-49cc-877c-dc28e872558f/exporter/0.log"
Sep 10 19:45:46.555357 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:46.555313    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77nq5_f1110bb8-b19b-49cc-877c-dc28e872558f/extractor/0.log"
Sep 10 19:45:48.085982 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:48.085950    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-w57fc"
Sep 10 19:45:49.766809 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:49.766753    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-xsg8v_11bd4506-ecf3-4b70-b37f-d39b75348869/manager/0.log"
Sep 10 19:45:49.842048 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:49.841949    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-xsg8v_11bd4506-ecf3-4b70-b37f-d39b75348869/copy-manifests/0.log"
Sep 10 19:45:51.011527 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:51.011502    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5f7cd94b-cqhc8_075a6585-5c1b-4f46-911d-2f7b26c45cb2/kube-auth-proxy/0.log"
Sep 10 19:45:51.390268 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:51.390237    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6748c6bd4b-rmpm8_6d33755e-8af6-4435-bba7-6b0f9cb6f17e/manager/0.log"
Sep 10 19:45:52.918803 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:52.918766    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5f7cd94b-cqhc8_075a6585-5c1b-4f46-911d-2f7b26c45cb2/kube-auth-proxy/0.log"
Sep 10 19:45:58.067387 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:58.067356    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5rfzq_0aa983d7-eafe-4eb0-9cf1-b236e928a649/kube-multus/0.log"
Sep 10 19:45:58.093875 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:58.093848    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-967gq_0f1754fb-fef4-4250-bffa-2032c9acdd11/kube-multus-additional-cni-plugins/0.log"
Sep 10 19:45:58.119755 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:58.119732    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-967gq_0f1754fb-fef4-4250-bffa-2032c9acdd11/egress-router-binary-copy/0.log"
Sep 10 19:45:58.145759 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:58.145735    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-967gq_0f1754fb-fef4-4250-bffa-2032c9acdd11/cni-plugins/0.log"
Sep 10 19:45:58.176696 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:58.176668    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-967gq_0f1754fb-fef4-4250-bffa-2032c9acdd11/bond-cni-plugin/0.log"
Sep 10 19:45:58.209522 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:58.209497    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-967gq_0f1754fb-fef4-4250-bffa-2032c9acdd11/routeoverride-cni/0.log"
Sep 10 19:45:58.232973 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:58.232949    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-967gq_0f1754fb-fef4-4250-bffa-2032c9acdd11/whereabouts-cni-bincopy/0.log"
Sep 10 19:45:58.260007 ip-10-0-141-45 kubenswrapper[2573]: I0910 19:45:58.259977    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-967gq_0f1754fb-fef4-4250-bffa-2032c9acdd11/whereabouts-cni/0.log"