Sep 10 15:30:17.741670 ip-10-0-134-9 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 15:30:17.741684 ip-10-0-134-9 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 15:30:17.741691 ip-10-0-134-9 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 15:30:17.741961 ip-10-0-134-9 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 15:30:27.753726 ip-10-0-134-9 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 15:30:27.753745 ip-10-0-134-9 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f2177c1d56af4b32a8173b6360e1a4cf -- Sep 10 15:32:19.651440 ip-10-0-134-9 systemd[1]: Starting Kubernetes Kubelet... Sep 10 15:32:20.152222 ip-10-0-134-9 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:32:20.152222 ip-10-0-134-9 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 15:32:20.152222 ip-10-0-134-9 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:32:20.152222 ip-10-0-134-9 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 15:32:20.152222 ip-10-0-134-9 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:32:20.155103 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.155003 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 15:32:20.162159 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162138 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:32:20.162159 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162155 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:32:20.162159 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162159 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:32:20.162159 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162162 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162167 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162170 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162173 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162176 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162179 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162182 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162185 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162188 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162191 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162194 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162197 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162200 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162202 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162205 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162207 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162210 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162214 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162218 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:32:20.162281 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162221 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162224 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162226 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162229 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162232 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162234 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162237 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162240 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162242 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162246 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162249 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162252 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162254 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162257 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162259 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162264 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162268 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162272 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162276 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:32:20.162751 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162279 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162282 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162285 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162289 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162292 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162295 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162297 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162300 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162302 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162305 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162308 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162310 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162313 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162316 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162318 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162321 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162323 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162326 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162328 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162331 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:32:20.163295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162334 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162336 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162339 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162341 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162344 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162346 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162350 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162353 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162356 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162359 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162361 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162364 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162366 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162369 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162372 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162375 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162378 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162381 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162383 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162386 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:32:20.163781 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162390 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162393 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162396 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162398 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162401 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162404 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162406 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162409 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162412 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162414 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162417 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162419 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162422 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162424 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162427 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162430 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162433 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.162435 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163173 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163181 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:32:20.164271 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163185 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163188 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163191 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163194 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163196 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163199 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163202 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163204 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163207 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163214 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163217 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163220 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163222 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163225 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163228 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163230 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163233 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163236 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163238 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163241 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:32:20.164774 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163243 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163246 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163248 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163251 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163254 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163256 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163259 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163261 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163264 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163267 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163270 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163272 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163276 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163279 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163282 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163286 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163289 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163293 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163296 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:32:20.165315 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163299 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163302 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163305 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163308 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163310 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163313 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163315 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163318 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163321 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163323 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163326 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163328 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163331 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163334 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163336 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163339 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163342 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163345 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163348 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163351 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:32:20.165797 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163353 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163356 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163359 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163361 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163364 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163367 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163370 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163373 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163376 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163378 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163381 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163384 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163387 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163389 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163392 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163394 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163397 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163400 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163403 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:32:20.166309 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163405 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163408 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163410 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163413 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163415 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163419 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163424 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163427 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163429 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163432 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163435 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163437 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163440 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163443 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163446 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163450 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163453 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163456 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163460 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163867 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:32:20.166770 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163872 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163875 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163879 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163882 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163884 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163887 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163890 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163893 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163895 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163903 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163906 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163909 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163911 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163914 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163916 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163919 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163921 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163924 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163927 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163929 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:32:20.167286 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163932 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163934 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163937 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163940 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163942 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163945 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163948 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163951 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163953 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163956 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163959 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163961 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163964 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163966 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163969 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163972 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163975 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163977 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163980 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163982 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163985 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:32:20.167772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163987 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163990 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163992 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163995 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.163998 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164001 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164003 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164006 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164008 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164011 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164013 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164016 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164022 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164025 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164028 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164030 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164033 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164036 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164039 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:32:20.168332 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164041 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164044 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164047 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164049 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164052 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164054 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164056 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164059 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164061 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164064 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164066 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164069 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164072 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164074 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164076 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164093 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164097 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164100 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164103 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164106 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:32:20.168803 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164109 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164112 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164114 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164117 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164120 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164124 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164127 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164129 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164132 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164134 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164137 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164141 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164144 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164147 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164151 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164154 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164157 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164160 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164243 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 15:32:20.169299 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164290 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164299 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164304 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164308 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164312 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164316 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164321 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164325 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164328 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164334 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164337 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164341 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164343 2580 flags.go:64] FLAG: --cgroup-root="" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164347 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164350 2580 flags.go:64] FLAG: --client-ca-file="" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164353 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164356 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164360 2580 flags.go:64] FLAG: --cluster-domain="" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164364 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164367 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164370 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164373 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164378 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 15:32:20.169749 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164381 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164384 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164388 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164391 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164394 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164397 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164400 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164403 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164407 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164410 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164413 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164416 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164419 2580 flags.go:64] FLAG: --enable-server="true" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164422 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164426 2580 flags.go:64] FLAG: --event-burst="100" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164429 2580 flags.go:64] FLAG: --event-qps="50" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164432 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164435 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164440 2580 flags.go:64] FLAG: --eviction-hard="" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164444 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164447 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164450 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164453 2580 flags.go:64] FLAG: --eviction-soft="" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164456 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164459 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 15:32:20.170322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164462 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164465 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164468 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164471 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164474 2580 flags.go:64] FLAG: --feature-gates="" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164478 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164481 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164484 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164487 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164490 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164493 2580 flags.go:64] FLAG: --help="false" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164496 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-134-9.ec2.internal" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164499 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164502 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164505 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164508 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164512 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164515 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164517 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164520 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164523 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164526 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164529 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164532 2580 flags.go:64] FLAG: --kube-reserved="" Sep 10 15:32:20.171053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164535 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164540 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164543 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164546 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164549 2580 flags.go:64] FLAG: --lock-file="" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164552 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164555 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164558 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164564 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164567 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164570 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164573 2580 flags.go:64] FLAG: --logging-format="text" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164576 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164579 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164582 2580 flags.go:64] FLAG: --manifest-url="" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164585 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164589 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164592 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164599 2580 flags.go:64] FLAG: --max-pods="110" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164602 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164605 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164608 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164611 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164614 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164617 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 15:32:20.171696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164620 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164627 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164630 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164633 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164636 2580 flags.go:64] FLAG: --pod-cidr="" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164639 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164645 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164648 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164651 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164655 2580 flags.go:64] FLAG: --port="10250" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164659 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164662 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f7ec3f15c73e1c59" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164666 2580 flags.go:64] FLAG: --qos-reserved="" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164669 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164672 2580 flags.go:64] FLAG: --register-node="true" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164675 2580 flags.go:64] FLAG: --register-with-taints="" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164678 2580 flags.go:64] FLAG: --registry-burst="10" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164681 2580 flags.go:64] FLAG: --registry-qps="5" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164684 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164687 2580 flags.go:64] FLAG: --reserved-memory="" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164691 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164694 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164697 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164700 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164703 2580 flags.go:64] FLAG: --runonce="false" Sep 10 15:32:20.172355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164706 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164708 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164712 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164715 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164718 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164720 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164724 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164727 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164730 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164733 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164736 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164738 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164742 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164745 2580 flags.go:64] FLAG: --system-cgroups="" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164748 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164753 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164757 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164760 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164764 2580 flags.go:64] FLAG: --tls-min-version="" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164767 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164770 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164773 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164776 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164780 2580 flags.go:64] FLAG: --v="2" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164792 2580 flags.go:64] FLAG: --version="false" Sep 10 15:32:20.172972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164798 2580 flags.go:64] FLAG: --vmodule="" Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164802 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.164806 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164947 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164951 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164954 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164957 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164961 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164964 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164967 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164970 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164973 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164976 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164978 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164981 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164983 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164986 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164989 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164991 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:32:20.173599 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164994 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164996 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.164999 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165002 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165006 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165009 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165012 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165014 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165017 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165020 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165023 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165026 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165029 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165032 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165035 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165037 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165040 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165043 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165045 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:32:20.174061 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165049 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165053 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165056 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165058 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165061 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165064 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165067 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165069 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165071 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165074 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165076 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165091 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165094 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165097 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165101 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165105 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165107 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165112 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165115 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:32:20.174565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165117 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165120 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165122 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165126 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165128 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165131 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165135 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165137 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165140 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165143 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165145 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165148 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165150 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165153 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165155 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165158 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165160 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165163 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165165 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165168 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:32:20.175033 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165170 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165173 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165175 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165178 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165181 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165183 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165186 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165189 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165191 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165193 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165197 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165200 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165203 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165205 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165208 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165211 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165214 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165216 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165221 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165224 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:32:20.175545 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165226 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165229 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165232 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165234 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.165237 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.165242 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.173400 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.173420 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.174567 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.174575 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174640 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174645 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174649 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174652 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174655 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:32:20.176060 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174658 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174661 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174663 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174666 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174669 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174671 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174674 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174677 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174679 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174683 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174685 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174688 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174690 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174693 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174695 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174698 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174701 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174703 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174706 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174709 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:32:20.176551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174711 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174714 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174717 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174720 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174730 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174732 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174735 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174738 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174740 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174743 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174746 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174748 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174751 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174755 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174760 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174763 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174766 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174769 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174772 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:32:20.177053 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174776 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174780 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174783 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174785 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174788 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174790 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174793 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174796 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174798 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174801 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174803 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174806 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174809 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174811 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174814 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174816 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174819 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174822 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174832 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174835 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:32:20.177534 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174837 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174840 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174843 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174846 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174848 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174851 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174853 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174856 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174858 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174861 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174864 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174866 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174869 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174871 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174874 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174877 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174879 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174882 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174884 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174887 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:32:20.178018 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174889 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174892 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174895 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174897 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174900 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174902 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174905 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174907 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174910 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174913 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174916 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174924 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174927 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174929 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:32:20.178580 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.174932 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.174937 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175104 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175110 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175113 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175116 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175119 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175121 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175125 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175127 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175130 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175132 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175135 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175137 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175140 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175143 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:32:20.178938 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175145 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175148 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175150 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175153 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175155 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175158 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175160 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175163 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175166 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175168 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175171 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175174 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175176 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175179 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175188 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175190 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175193 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175196 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175199 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175201 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:32:20.179357 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175204 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175206 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175209 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175211 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175214 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175217 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175219 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175222 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175224 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175227 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175229 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175232 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175234 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175237 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175239 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175242 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175244 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175247 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175249 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175252 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:32:20.180014 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175255 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175257 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175260 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175263 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175267 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175271 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175274 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175283 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175286 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175289 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175292 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175294 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175297 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175301 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175304 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175307 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175310 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175313 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:32:20.180565 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175316 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175318 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175321 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175324 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175326 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175329 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175331 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175340 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175342 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175345 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175347 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175350 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175353 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175355 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175358 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175361 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175364 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175366 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175369 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175371 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:32:20.181022 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175374 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175376 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175384 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175387 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175389 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175392 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:20.175394 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.175399 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.175530 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.178050 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.179485 2580 server.go:1019] "Starting client certificate rotation" Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.179609 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 15:32:20.181540 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.179655 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 15:32:20.185405 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.185385 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 15:32:20.192123 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.192101 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 15:32:20.207185 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.207167 2580 log.go:25] "Validated CRI v1 runtime API" Sep 10 15:32:20.210776 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.210756 2580 log.go:25] "Validated CRI v1 image API" Sep 10 15:32:20.211643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.211625 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 15:32:20.213782 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.213765 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 15:32:20.219970 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.219944 2580 fs.go:135] Filesystem UUIDs: map[6bc3d0f5-fd02-4a10-9b7f-07ee39bec463:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bc766b0e-4a55-4b46-b81c-045900c7b1d1:/dev/nvme0n1p3] Sep 10 15:32:20.220052 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.219968 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 15:32:20.224274 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.224164 2580 manager.go:216] Machine: {Timestamp:2026-09-10 15:32:20.22368977 +0000 UTC m=+0.441337763 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099111 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2be2fb446d9ad3e22e22e6053b05c9 SystemUUID:ec2be2fb-446d-9ad3-e22e-22e6053b05c9 BootID:f2177c1d-56af-4b32-a817-3b6360e1a4cf Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e5:37:27:da:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e5:37:27:da:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:e5:dc:ef:50:b0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 15:32:20.224274 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.224271 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 15:32:20.224375 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.224347 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 15:32:20.228503 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.228480 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 15:32:20.228642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.228505 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 15:32:20.229331 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.229321 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 15:32:20.229362 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.229334 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 15:32:20.229362 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.229343 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 15:32:20.230531 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.230520 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 15:32:20.232434 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.232424 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 15:32:20.241625 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.241610 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 10 15:32:20.241762 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.241752 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 15:32:20.241799 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.241783 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 10 15:32:20.241799 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.241792 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 15:32:20.241860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.241806 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 15:32:20.241860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.241816 2580 kubelet.go:398] "Adding apiserver pod source" Sep 10 15:32:20.241860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.241825 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 15:32:20.242275 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.242259 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 15:32:20.242346 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.242289 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 15:32:20.244305 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.244271 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t7hmp" Sep 10 15:32:20.244844 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.244828 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 15:32:20.247410 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.247394 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 15:32:20.247495 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.247433 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 15:32:20.248316 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248304 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 15:32:20.248380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248327 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 15:32:20.248380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248335 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 15:32:20.248380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248344 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 15:32:20.248380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248357 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 15:32:20.248380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248366 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 15:32:20.248576 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248453 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 15:32:20.248576 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248463 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 15:32:20.248576 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248474 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 15:32:20.248576 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248483 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 15:32:20.248576 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248506 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 15:32:20.248576 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.248520 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 15:32:20.250832 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.250809 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 15:32:20.251005 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.250983 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 15:32:20.255663 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.255645 2580 server.go:1268] "Started kubelet" Sep 10 15:32:20.256217 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.256197 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 15:32:20.256584 ip-10-0-134-9 systemd[1]: Started Kubernetes Kubelet. Sep 10 15:32:20.258521 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.258412 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 15:32:20.258589 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.258569 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 15:32:20.259209 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.259189 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t7hmp" Sep 10 15:32:20.260580 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.260561 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 15:32:20.261217 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.261188 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-9.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 15:32:20.261376 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.261350 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 15:32:20.261425 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.261370 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 15:32:20.261546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.261534 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 10 15:32:20.262231 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.262202 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 15:32:20.265793 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.265777 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 15:32:20.266392 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.266374 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 15:32:20.266513 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.266478 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:20.267393 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267374 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 15:32:20.267393 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267393 2580 factory.go:55] Registering systemd factory Sep 10 15:32:20.267584 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267402 2580 factory.go:223] Registration of the systemd container factory successfully Sep 10 15:32:20.267642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267621 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 15:32:20.267642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267622 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 15:32:20.267740 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267641 2580 factory.go:153] Registering CRI-O factory Sep 10 15:32:20.267740 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267651 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 15:32:20.267740 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267655 2580 factory.go:223] Registration of the crio container factory successfully Sep 10 15:32:20.267740 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267672 2580 factory.go:103] Registering Raw factory Sep 10 15:32:20.267740 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267684 2580 manager.go:1196] Started watching for new ooms in manager Sep 10 15:32:20.267964 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267745 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 10 15:32:20.267964 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.267755 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 10 15:32:20.268044 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.268039 2580 manager.go:349] Starting recovery of all containers Sep 10 15:32:20.269847 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.267769 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-9.ec2.internal.18d3ff7192ce69c6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-9.ec2.internal,UID:ip-10-0-134-9.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-9.ec2.internal,},FirstTimestamp:2026-09-10 15:32:20.255607238 +0000 UTC m=+0.473255230,LastTimestamp:2026-09-10 15:32:20.255607238 +0000 UTC m=+0.473255230,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-9.ec2.internal,}" Sep 10 15:32:20.275543 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.275524 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:32:20.278525 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.278494 2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-9.ec2.internal\" not found" node="ip-10-0-134-9.ec2.internal" Sep 10 15:32:20.279651 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.279505 2580 manager.go:354] Recovery completed Sep 10 15:32:20.285118 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.285101 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:32:20.286956 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.286937 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:32:20.287034 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.286973 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:32:20.287034 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.286989 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:32:20.287578 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.287564 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 15:32:20.287645 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.287578 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 15:32:20.287645 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.287598 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 10 15:32:20.287645 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.287598 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:32:20.288335 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.288318 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:32:20.288427 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.288346 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:32:20.288427 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.288359 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:32:20.289139 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.289126 2580 policy_none.go:49] "None policy: Start" Sep 10 15:32:20.289184 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.289144 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 15:32:20.289184 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.289154 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 15:32:20.290443 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.290432 2580 policy_none.go:47] "Start" Sep 10 15:32:20.364966 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.364926 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 15:32:20.366293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.366271 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 15:32:20.366393 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.366324 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 15:32:20.366393 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.366353 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 15:32:20.366499 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.366442 2580 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 10 15:32:20.366671 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.366630 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:20.368991 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.368969 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:32:20.378684 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.378668 2580 manager.go:337] "Starting Device Plugin manager" Sep 10 15:32:20.378785 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.378707 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 15:32:20.378785 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.378717 2580 server.go:87] "Starting device plugin registration server" Sep 10 15:32:20.379056 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.379043 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 15:32:20.379162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.379056 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 15:32:20.379217 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.379206 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 15:32:20.379305 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.379292 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 15:32:20.379305 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.379305 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 15:32:20.379776 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.379746 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 15:32:20.379860 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.379800 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:20.467295 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.467250 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal"] Sep 10 15:32:20.467425 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.467346 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:32:20.468689 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.468663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3cdfc09abde163e5efcd6f1fa3f633fc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" (UID: \"3cdfc09abde163e5efcd6f1fa3f633fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Sep 10 15:32:20.470043 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.470027 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:32:20.470127 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.470057 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:32:20.470127 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.470067 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:32:20.471501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.471488 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:32:20.471606 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.471587 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Sep 10 15:32:20.471684 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.471624 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:32:20.472224 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.472208 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:32:20.472299 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.472237 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:32:20.472299 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.472249 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:32:20.472299 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.472217 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:32:20.472299 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.472279 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:32:20.472299 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.472292 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:32:20.473699 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.473680 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Sep 10 15:32:20.473794 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.473710 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:32:20.474442 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.474426 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:32:20.474524 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.474455 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:32:20.474524 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.474471 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:32:20.479194 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.479180 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:32:20.479847 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.479826 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:32:20.479938 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.479852 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:32:20.479938 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.479867 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:32:20.479938 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.479890 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-9.ec2.internal" Sep 10 15:32:20.488105 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:20.488078 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-9.ec2.internal" Sep 10 15:32:20.488158 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.488112 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-9.ec2.internal\": node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:20.495660 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.495643 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-9.ec2.internal\" not found" node="ip-10-0-134-9.ec2.internal" Sep 10 15:32:20.499545 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.499529 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-9.ec2.internal\" not found" node="ip-10-0-134-9.ec2.internal" Sep 10 15:32:20.501978 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:20.501959 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:22.224226 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.224194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3cdfc09abde163e5efcd6f1fa3f633fc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" (UID: \"3cdfc09abde163e5efcd6f1fa3f633fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.224226 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.224230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d99c6e4cbc478f4080fa6c464fe6df80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"d99c6e4cbc478f4080fa6c464fe6df80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.224393 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.224254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d99c6e4cbc478f4080fa6c464fe6df80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"d99c6e4cbc478f4080fa6c464fe6df80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.224393 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.224323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3cdfc09abde163e5efcd6f1fa3f633fc-config\") pod \"kube-apiserver-proxy-ip-10-0-134-9.ec2.internal\" (UID: \"3cdfc09abde163e5efcd6f1fa3f633fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.257379 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:22.257351 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:22.325099 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.325069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d99c6e4cbc478f4080fa6c464fe6df80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"d99c6e4cbc478f4080fa6c464fe6df80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.325204 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.325103 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d99c6e4cbc478f4080fa6c464fe6df80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"d99c6e4cbc478f4080fa6c464fe6df80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.325204 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.325145 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d99c6e4cbc478f4080fa6c464fe6df80-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"d99c6e4cbc478f4080fa6c464fe6df80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.325204 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.325192 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d99c6e4cbc478f4080fa6c464fe6df80-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal\" (UID: \"d99c6e4cbc478f4080fa6c464fe6df80\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.358376 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:22.358339 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:22.454114 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.454046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.458482 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.458321 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Sep 10 15:32:22.459400 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:22.459379 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:22.560158 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:22.560095 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:22.660642 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:22.660610 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:22.761380 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:22.761301 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:22.833784 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.833754 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 15:32:22.834261 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.833886 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 15:32:22.834261 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.833913 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 15:32:22.862064 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:22.862039 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:22.922791 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.922746 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:27:21 +0000 UTC" deadline="2028-04-03 15:30:06.714157153 +0000 UTC" Sep 10 15:32:22.922791 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:22.922789 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13703h57m43.791372411s" Sep 10 15:32:22.962328 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:22.962299 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:23.016730 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:23.016696 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99c6e4cbc478f4080fa6c464fe6df80.slice/crio-9b2c9d1382ae621b2ce2008c4929d8555da81aafb8caf7e79332e96556801e19 WatchSource:0}: Error finding container 9b2c9d1382ae621b2ce2008c4929d8555da81aafb8caf7e79332e96556801e19: Status 404 returned error can't find the container with id 9b2c9d1382ae621b2ce2008c4929d8555da81aafb8caf7e79332e96556801e19 Sep 10 15:32:23.017144 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:23.017126 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdfc09abde163e5efcd6f1fa3f633fc.slice/crio-af7f468a638a9b19e76bdd592f8dd98dcfd5b2318daddbfed882dd926e5dbbd2 WatchSource:0}: Error finding container af7f468a638a9b19e76bdd592f8dd98dcfd5b2318daddbfed882dd926e5dbbd2: Status 404 returned error can't find the container with id af7f468a638a9b19e76bdd592f8dd98dcfd5b2318daddbfed882dd926e5dbbd2 Sep 10 15:32:23.023565 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.023547 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:32:23.025253 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.025209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" event={"ID":"d99c6e4cbc478f4080fa6c464fe6df80","Type":"ContainerStarted","Data":"9b2c9d1382ae621b2ce2008c4929d8555da81aafb8caf7e79332e96556801e19"} Sep 10 15:32:23.026409 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.026386 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" event={"ID":"3cdfc09abde163e5efcd6f1fa3f633fc","Type":"ContainerStarted","Data":"af7f468a638a9b19e76bdd592f8dd98dcfd5b2318daddbfed882dd926e5dbbd2"} Sep 10 15:32:23.062400 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:23.062376 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:23.162929 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:23.162903 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:23.263526 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:23.263464 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:23.364153 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:23.364116 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-9.ec2.internal\" not found" Sep 10 15:32:23.419762 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.419735 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:32:23.454524 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.454495 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:32:23.522095 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.522005 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" Sep 10 15:32:23.530923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.530885 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 15:32:23.532015 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.531992 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" Sep 10 15:32:23.540734 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.540712 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 15:32:23.898711 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.898677 2580 apiserver.go:52] "Watching apiserver" Sep 10 15:32:23.905037 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.905005 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 15:32:23.905475 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.905446 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rzbl8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b","openshift-cluster-node-tuning-operator/tuned-mtjxt","openshift-dns/node-resolver-hvffx","openshift-multus/multus-additional-cni-plugins-xb459","openshift-multus/multus-jh65b","openshift-multus/network-metrics-daemon-hvz5r","openshift-network-diagnostics/network-check-target-q8p4v","kube-system/global-pull-secret-syncer-8ld8d","kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal","openshift-image-registry/node-ca-6hvnc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal","openshift-network-operator/iptables-alerter-rbdsp","openshift-ovn-kubernetes/ovnkube-node-g68fj"] Sep 10 15:32:23.907931 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.907489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:23.907931 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:23.907575 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:23.908931 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.908796 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:23.910355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.910286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.911617 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.910951 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 15:32:23.911617 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.911004 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:32:23.911617 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.911062 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m4rc6\"" Sep 10 15:32:23.911617 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.911201 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 15:32:23.911883 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.911865 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:23.912655 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.912619 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zzx8r\"" Sep 10 15:32:23.912847 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.912813 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 15:32:23.913134 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.913114 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:32:23.914242 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.913813 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 15:32:23.914242 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.913913 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:23.914242 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.913994 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mcqq5\"" Sep 10 15:32:23.914694 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.914463 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 15:32:23.915850 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.915832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.916012 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.915987 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 15:32:23.916256 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.916235 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 15:32:23.916256 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.916246 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 15:32:23.916766 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.916549 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 15:32:23.916766 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.916622 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 15:32:23.916766 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.916678 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6f4kq\"" Sep 10 15:32:23.917152 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.917135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:23.917312 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:23.917291 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:23.918205 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.918067 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 15:32:23.918301 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.918251 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ktlhh\"" Sep 10 15:32:23.918510 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.918489 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:23.921415 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.921392 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 15:32:23.921522 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.921396 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:23.921522 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.921513 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-prxxk\"" Sep 10 15:32:23.921627 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.921522 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 15:32:23.922485 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.922227 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 15:32:23.924055 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.924034 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ptkmb\"" Sep 10 15:32:23.924475 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.924438 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 15:32:23.924659 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.924513 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 15:32:23.924819 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.924411 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 15:32:23.924966 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.924946 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.926909 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.926890 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 15:32:23.927193 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.927176 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 15:32:23.927480 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.927464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:23.928071 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.928011 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 15:32:23.928157 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.928076 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rz49d\"" Sep 10 15:32:23.928157 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.928144 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 15:32:23.928257 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.928146 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 15:32:23.928873 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.928857 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 15:32:23.929574 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.929330 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:23.929574 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:23.929394 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:23.930252 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.930233 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 15:32:23.930252 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.930245 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 15:32:23.930561 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.930542 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p7v6p\"" Sep 10 15:32:23.930759 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.930737 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 15:32:23.932388 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932365 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/368dd85e-db5c-42da-b98c-b8cd54960d1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:23.932482 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.932482 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f636588a-67d1-4196-91d5-783f5ca11b8a-hosts-file\") pod \"node-resolver-hvffx\" (UID: \"f636588a-67d1-4196-91d5-783f5ca11b8a\") " pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:23.932554 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:23.932554 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50bb48bf-062a-43c9-b426-ab56a1bb466f-tmp\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.932554 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-socket-dir-parent\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.932679 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:23.932679 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-cni-bin\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.932679 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932614 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbrx\" (UniqueName: \"kubernetes.io/projected/2efed680-b002-4d32-8d04-33de60e395e4-kube-api-access-lbbrx\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.932679 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfchx\" (UniqueName: \"kubernetes.io/projected/6150e082-626a-47a0-b8a0-5a9ea64b8251-kube-api-access-rfchx\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:23.932860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-kubelet\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.932860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xml7q\" (UniqueName: \"kubernetes.io/projected/af249678-090c-4a60-b691-634f0c9c131c-kube-api-access-xml7q\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:23.932860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-sysctl-conf\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.932860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-lib-modules\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.933041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fed4497c-aa2c-4f87-a183-ac8d193d9e93-agent-certs\") pod \"konnectivity-agent-rzbl8\" (UID: \"fed4497c-aa2c-4f87-a183-ac8d193d9e93\") " pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:23.933041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-socket-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:23.933041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-registration-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:23.933041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2n6\" (UniqueName: \"kubernetes.io/projected/50bb48bf-062a-43c9-b426-ab56a1bb466f-kube-api-access-6g2n6\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.933041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.932983 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/368dd85e-db5c-42da-b98c-b8cd54960d1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:23.933041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-etc-openvswitch\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.933290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25d697d5-2458-4160-9e16-3fcef346035d-iptables-alerter-script\") pod \"iptables-alerter-rbdsp\" (UID: \"25d697d5-2458-4160-9e16-3fcef346035d\") " pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:23.933290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933083 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25d697d5-2458-4160-9e16-3fcef346035d-host-slash\") pod \"iptables-alerter-rbdsp\" (UID: \"25d697d5-2458-4160-9e16-3fcef346035d\") " pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:23.933290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-device-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:23.933290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933149 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/368dd85e-db5c-42da-b98c-b8cd54960d1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:23.933290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-cnibin\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.933290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baa7bbbd-ba07-41eb-883f-4fa765f40091-cni-binary-copy\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.933290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-run-netns\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.933290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-var-lib-kubelet\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.933290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-etc-kubernetes\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-dbus\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-modprobe-d\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-run\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-var-lib-cni-bin\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nrv\" (UniqueName: \"kubernetes.io/projected/baa7bbbd-ba07-41eb-883f-4fa765f40091-kube-api-access-z6nrv\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-run-openvswitch\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2efed680-b002-4d32-8d04-33de60e395e4-ovnkube-config\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppsw\" (UniqueName: \"kubernetes.io/projected/f636588a-67d1-4196-91d5-783f5ca11b8a-kube-api-access-wppsw\") pod \"node-resolver-hvffx\" (UID: \"f636588a-67d1-4196-91d5-783f5ca11b8a\") " pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-run-systemd\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/368dd85e-db5c-42da-b98c-b8cd54960d1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-cni-dir\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-run-k8s-cni-cncf-io\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-hostroot\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f636588a-67d1-4196-91d5-783f5ca11b8a-tmp-dir\") pod \"node-resolver-hvffx\" (UID: \"f636588a-67d1-4196-91d5-783f5ca11b8a\") " pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-sys-fs\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:23.933718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-system-cni-dir\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-kubelet-config\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2efed680-b002-4d32-8d04-33de60e395e4-env-overrides\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933797 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-run-netns\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fed4497c-aa2c-4f87-a183-ac8d193d9e93-konnectivity-ca\") pod \"konnectivity-agent-rzbl8\" (UID: \"fed4497c-aa2c-4f87-a183-ac8d193d9e93\") " pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-systemd\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-tuned\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.933948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-var-lib-openvswitch\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-run-ovn\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2efed680-b002-4d32-8d04-33de60e395e4-ovnkube-script-lib\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkrrq\" (UniqueName: \"kubernetes.io/projected/25d697d5-2458-4160-9e16-3fcef346035d-kube-api-access-jkrrq\") pod \"iptables-alerter-rbdsp\" (UID: \"25d697d5-2458-4160-9e16-3fcef346035d\") " pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-var-lib-kubelet\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934125 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-slash\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934148 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-log-socket\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934173 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-cni-netd\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934197 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.934505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2efed680-b002-4d32-8d04-33de60e395e4-ovn-node-metrics-cert\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlttp\" (UniqueName: \"kubernetes.io/projected/368dd85e-db5c-42da-b98c-b8cd54960d1b-kube-api-access-zlttp\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-conf-dir\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-node-log\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934335 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-sysconfig\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-kubernetes\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934439 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-sys\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-host\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-var-lib-cni-multus\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-daemon-config\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934564 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-run-multus-certs\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-systemd-units\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-sysctl-d\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934649 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/368dd85e-db5c-42da-b98c-b8cd54960d1b-system-cni-dir\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:23.935293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.934673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/368dd85e-db5c-42da-b98c-b8cd54960d1b-cnibin\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:23.955691 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.955669 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:32:23.956795 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.956774 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ntp22" Sep 10 15:32:23.964889 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:23.964866 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ntp22" Sep 10 15:32:24.026022 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.025158 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 15:32:24.034974 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.034935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-systemd\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.034982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-tuned\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-var-lib-openvswitch\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-run-ovn\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2efed680-b002-4d32-8d04-33de60e395e4-ovnkube-script-lib\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkrrq\" (UniqueName: \"kubernetes.io/projected/25d697d5-2458-4160-9e16-3fcef346035d-kube-api-access-jkrrq\") pod \"iptables-alerter-rbdsp\" (UID: \"25d697d5-2458-4160-9e16-3fcef346035d\") " pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-var-lib-kubelet\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-slash\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-systemd\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035127 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-var-lib-openvswitch\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-log-socket\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-log-socket\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-cni-netd\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-var-lib-kubelet\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-cni-netd\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-run-ovn\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2efed680-b002-4d32-8d04-33de60e395e4-ovn-node-metrics-cert\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlttp\" (UniqueName: \"kubernetes.io/projected/368dd85e-db5c-42da-b98c-b8cd54960d1b-kube-api-access-zlttp\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035285 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-conf-dir\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035306 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-slash\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-node-log\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcq96\" (UniqueName: \"kubernetes.io/projected/b92f14a2-c880-4843-98be-7a8480914160-kube-api-access-mcq96\") pod \"node-ca-6hvnc\" (UID: \"b92f14a2-c880-4843-98be-7a8480914160\") " pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-node-log\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.035618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-sysconfig\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-conf-dir\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-kubernetes\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-sys\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035560 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-sysconfig\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-sys\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-host\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035669 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-host\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-kubernetes\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035717 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-var-lib-cni-multus\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2efed680-b002-4d32-8d04-33de60e395e4-ovnkube-script-lib\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-daemon-config\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035765 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-var-lib-cni-multus\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-run-multus-certs\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035776 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035805 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-run-multus-certs\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-systemd-units\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.036389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-systemd-units\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-sysctl-d\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/368dd85e-db5c-42da-b98c-b8cd54960d1b-system-cni-dir\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/368dd85e-db5c-42da-b98c-b8cd54960d1b-cnibin\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/368dd85e-db5c-42da-b98c-b8cd54960d1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.035990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036013 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f636588a-67d1-4196-91d5-783f5ca11b8a-hosts-file\") pod \"node-resolver-hvffx\" (UID: \"f636588a-67d1-4196-91d5-783f5ca11b8a\") " pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50bb48bf-062a-43c9-b426-ab56a1bb466f-tmp\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-socket-dir-parent\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-cni-bin\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbrx\" (UniqueName: \"kubernetes.io/projected/2efed680-b002-4d32-8d04-33de60e395e4-kube-api-access-lbbrx\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b92f14a2-c880-4843-98be-7a8480914160-host\") pod \"node-ca-6hvnc\" (UID: \"b92f14a2-c880-4843-98be-7a8480914160\") " pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfchx\" (UniqueName: \"kubernetes.io/projected/6150e082-626a-47a0-b8a0-5a9ea64b8251-kube-api-access-rfchx\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036253 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-daemon-config\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-kubelet\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.037106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xml7q\" (UniqueName: \"kubernetes.io/projected/af249678-090c-4a60-b691-634f0c9c131c-kube-api-access-xml7q\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b92f14a2-c880-4843-98be-7a8480914160-serviceca\") pod \"node-ca-6hvnc\" (UID: \"b92f14a2-c880-4843-98be-7a8480914160\") " pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-socket-dir-parent\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-sysctl-conf\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-run-ovn-kubernetes\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-lib-modules\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/368dd85e-db5c-42da-b98c-b8cd54960d1b-system-cni-dir\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f636588a-67d1-4196-91d5-783f5ca11b8a-hosts-file\") pod \"node-resolver-hvffx\" (UID: \"f636588a-67d1-4196-91d5-783f5ca11b8a\") " pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fed4497c-aa2c-4f87-a183-ac8d193d9e93-agent-certs\") pod \"konnectivity-agent-rzbl8\" (UID: \"fed4497c-aa2c-4f87-a183-ac8d193d9e93\") " pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-socket-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-registration-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2n6\" (UniqueName: \"kubernetes.io/projected/50bb48bf-062a-43c9-b426-ab56a1bb466f-kube-api-access-6g2n6\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-sysctl-d\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/368dd85e-db5c-42da-b98c-b8cd54960d1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-etc-openvswitch\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036531 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/368dd85e-db5c-42da-b98c-b8cd54960d1b-cnibin\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25d697d5-2458-4160-9e16-3fcef346035d-iptables-alerter-script\") pod \"iptables-alerter-rbdsp\" (UID: \"25d697d5-2458-4160-9e16-3fcef346035d\") " pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:24.037856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-etc-selinux\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25d697d5-2458-4160-9e16-3fcef346035d-host-slash\") pod \"iptables-alerter-rbdsp\" (UID: \"25d697d5-2458-4160-9e16-3fcef346035d\") " pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036610 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-etc-openvswitch\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-device-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/368dd85e-db5c-42da-b98c-b8cd54960d1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036652 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/368dd85e-db5c-42da-b98c-b8cd54960d1b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-cnibin\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-registration-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036767 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-cnibin\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036860 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-socket-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baa7bbbd-ba07-41eb-883f-4fa765f40091-cni-binary-copy\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-run-netns\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-var-lib-kubelet\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-etc-kubernetes\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.036988 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-dbus\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-modprobe-d\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-run\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.038642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-var-lib-cni-bin\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nrv\" (UniqueName: \"kubernetes.io/projected/baa7bbbd-ba07-41eb-883f-4fa765f40091-kube-api-access-z6nrv\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-run-openvswitch\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25d697d5-2458-4160-9e16-3fcef346035d-iptables-alerter-script\") pod \"iptables-alerter-rbdsp\" (UID: \"25d697d5-2458-4160-9e16-3fcef346035d\") " pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2efed680-b002-4d32-8d04-33de60e395e4-ovnkube-config\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppsw\" (UniqueName: \"kubernetes.io/projected/f636588a-67d1-4196-91d5-783f5ca11b8a-kube-api-access-wppsw\") pod \"node-resolver-hvffx\" (UID: \"f636588a-67d1-4196-91d5-783f5ca11b8a\") " pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-run-systemd\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/368dd85e-db5c-42da-b98c-b8cd54960d1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-cni-dir\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-run-k8s-cni-cncf-io\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-hostroot\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.037344 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f636588a-67d1-4196-91d5-783f5ca11b8a-tmp-dir\") pod \"node-resolver-hvffx\" (UID: \"f636588a-67d1-4196-91d5-783f5ca11b8a\") " pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-sys-fs\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-system-cni-dir\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-kubelet-config\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:24.039413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2efed680-b002-4d32-8d04-33de60e395e4-env-overrides\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.037509 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret podName:f044f5a4-1d02-48b3-a5c3-0dc9f2212f51 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:24.537477775 +0000 UTC m=+3.099786963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret") pod "global-pull-secret-syncer-8ld8d" (UID: "f044f5a4-1d02-48b3-a5c3-0dc9f2212f51") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037543 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-run-netns\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-run-netns\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2efed680-b002-4d32-8d04-33de60e395e4-ovnkube-config\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-cni-bin\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2efed680-b002-4d32-8d04-33de60e395e4-env-overrides\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037955 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25d697d5-2458-4160-9e16-3fcef346035d-host-slash\") pod \"iptables-alerter-rbdsp\" (UID: \"25d697d5-2458-4160-9e16-3fcef346035d\") " pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.037997 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/368dd85e-db5c-42da-b98c-b8cd54960d1b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038006 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-host-kubelet\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fed4497c-aa2c-4f87-a183-ac8d193d9e93-konnectivity-ca\") pod \"konnectivity-agent-rzbl8\" (UID: \"fed4497c-aa2c-4f87-a183-ac8d193d9e93\") " pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038060 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-device-dir\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038100 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-run-netns\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-var-lib-kubelet\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038148 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-etc-kubernetes\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038210 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-sysctl-conf\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-lib-modules\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.040143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-dbus\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/368dd85e-db5c-42da-b98c-b8cd54960d1b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038466 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-modprobe-d\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50bb48bf-062a-43c9-b426-ab56a1bb466f-run\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fed4497c-aa2c-4f87-a183-ac8d193d9e93-konnectivity-ca\") pod \"konnectivity-agent-rzbl8\" (UID: \"fed4497c-aa2c-4f87-a183-ac8d193d9e93\") " pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-var-lib-cni-bin\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038605 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-run-openvswitch\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038654 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-hostroot\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.038717 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.038772 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs podName:6150e082-626a-47a0-b8a0-5a9ea64b8251 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:24.538756578 +0000 UTC m=+3.101065758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs") pod "network-metrics-daemon-hvz5r" (UID: "6150e082-626a-47a0-b8a0-5a9ea64b8251") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-system-cni-dir\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038807 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2efed680-b002-4d32-8d04-33de60e395e4-run-systemd\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/af249678-090c-4a60-b691-634f0c9c131c-sys-fs\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-kubelet-config\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038938 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-multus-cni-dir\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038951 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/368dd85e-db5c-42da-b98c-b8cd54960d1b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038062 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/baa7bbbd-ba07-41eb-883f-4fa765f40091-cni-binary-copy\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.040862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/baa7bbbd-ba07-41eb-883f-4fa765f40091-host-run-k8s-cni-cncf-io\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.041705 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.038990 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f636588a-67d1-4196-91d5-783f5ca11b8a-tmp-dir\") pod \"node-resolver-hvffx\" (UID: \"f636588a-67d1-4196-91d5-783f5ca11b8a\") " pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:24.041705 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.039889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50bb48bf-062a-43c9-b426-ab56a1bb466f-tmp\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.041705 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.040162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fed4497c-aa2c-4f87-a183-ac8d193d9e93-agent-certs\") pod \"konnectivity-agent-rzbl8\" (UID: \"fed4497c-aa2c-4f87-a183-ac8d193d9e93\") " pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:24.041705 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.040403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2efed680-b002-4d32-8d04-33de60e395e4-ovn-node-metrics-cert\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.041705 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.041406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50bb48bf-062a-43c9-b426-ab56a1bb466f-etc-tuned\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.043433 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.043397 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlttp\" (UniqueName: \"kubernetes.io/projected/368dd85e-db5c-42da-b98c-b8cd54960d1b-kube-api-access-zlttp\") pod \"multus-additional-cni-plugins-xb459\" (UID: \"368dd85e-db5c-42da-b98c-b8cd54960d1b\") " pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.043774 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.043732 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkrrq\" (UniqueName: \"kubernetes.io/projected/25d697d5-2458-4160-9e16-3fcef346035d-kube-api-access-jkrrq\") pod \"iptables-alerter-rbdsp\" (UID: \"25d697d5-2458-4160-9e16-3fcef346035d\") " pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:24.045679 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.045655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfchx\" (UniqueName: \"kubernetes.io/projected/6150e082-626a-47a0-b8a0-5a9ea64b8251-kube-api-access-rfchx\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:24.046150 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.046129 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2n6\" (UniqueName: \"kubernetes.io/projected/50bb48bf-062a-43c9-b426-ab56a1bb466f-kube-api-access-6g2n6\") pod \"tuned-mtjxt\" (UID: \"50bb48bf-062a-43c9-b426-ab56a1bb466f\") " pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.048833 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.048803 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xml7q\" (UniqueName: \"kubernetes.io/projected/af249678-090c-4a60-b691-634f0c9c131c-kube-api-access-xml7q\") pod \"aws-ebs-csi-driver-node-mgl2b\" (UID: \"af249678-090c-4a60-b691-634f0c9c131c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.048926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.048836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbrx\" (UniqueName: \"kubernetes.io/projected/2efed680-b002-4d32-8d04-33de60e395e4-kube-api-access-lbbrx\") pod \"ovnkube-node-g68fj\" (UID: \"2efed680-b002-4d32-8d04-33de60e395e4\") " pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.056022 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.055987 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppsw\" (UniqueName: \"kubernetes.io/projected/f636588a-67d1-4196-91d5-783f5ca11b8a-kube-api-access-wppsw\") pod \"node-resolver-hvffx\" (UID: \"f636588a-67d1-4196-91d5-783f5ca11b8a\") " pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:24.057253 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.057232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nrv\" (UniqueName: \"kubernetes.io/projected/baa7bbbd-ba07-41eb-883f-4fa765f40091-kube-api-access-z6nrv\") pod \"multus-jh65b\" (UID: \"baa7bbbd-ba07-41eb-883f-4fa765f40091\") " pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.138529 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.138495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b92f14a2-c880-4843-98be-7a8480914160-host\") pod \"node-ca-6hvnc\" (UID: \"b92f14a2-c880-4843-98be-7a8480914160\") " pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.138529 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.138532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b92f14a2-c880-4843-98be-7a8480914160-serviceca\") pod \"node-ca-6hvnc\" (UID: \"b92f14a2-c880-4843-98be-7a8480914160\") " pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.138770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.138614 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b92f14a2-c880-4843-98be-7a8480914160-host\") pod \"node-ca-6hvnc\" (UID: \"b92f14a2-c880-4843-98be-7a8480914160\") " pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.138770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.138707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcq96\" (UniqueName: \"kubernetes.io/projected/b92f14a2-c880-4843-98be-7a8480914160-kube-api-access-mcq96\") pod \"node-ca-6hvnc\" (UID: \"b92f14a2-c880-4843-98be-7a8480914160\") " pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.138770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.138758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:24.139691 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.139670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b92f14a2-c880-4843-98be-7a8480914160-serviceca\") pod \"node-ca-6hvnc\" (UID: \"b92f14a2-c880-4843-98be-7a8480914160\") " pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.144650 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.144628 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:32:24.144650 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.144651 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:32:24.144802 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.144667 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8mfr9 for pod openshift-network-diagnostics/network-check-target-q8p4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:24.144802 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.144734 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9 podName:8d47d830-d5d0-45ff-92d5-c605bd23d46f nodeName:}" failed. No retries permitted until 2026-09-10 15:32:24.644720291 +0000 UTC m=+3.207029460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8mfr9" (UniqueName: "kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9") pod "network-check-target-q8p4v" (UID: "8d47d830-d5d0-45ff-92d5-c605bd23d46f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:24.147117 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.147088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcq96\" (UniqueName: \"kubernetes.io/projected/b92f14a2-c880-4843-98be-7a8480914160-kube-api-access-mcq96\") pod \"node-ca-6hvnc\" (UID: \"b92f14a2-c880-4843-98be-7a8480914160\") " pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.224701 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.224620 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rbdsp" Sep 10 15:32:24.234231 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.234205 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" Sep 10 15:32:24.243435 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.243399 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hvffx" Sep 10 15:32:24.248826 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.248807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xb459" Sep 10 15:32:24.256087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.256052 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jh65b" Sep 10 15:32:24.262865 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.262846 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:24.269671 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.269649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" Sep 10 15:32:24.277458 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.277435 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:24.283140 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.283118 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6hvnc" Sep 10 15:32:24.435624 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.435592 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:32:24.541236 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.541151 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:24.541236 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.541220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:24.541438 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.541321 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:24.541438 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.541379 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs podName:6150e082-626a-47a0-b8a0-5a9ea64b8251 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:25.541366459 +0000 UTC m=+4.103675626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs") pod "network-metrics-daemon-hvz5r" (UID: "6150e082-626a-47a0-b8a0-5a9ea64b8251") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:24.541438 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.541321 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:24.541551 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.541478 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret podName:f044f5a4-1d02-48b3-a5c3-0dc9f2212f51 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:25.541462899 +0000 UTC m=+4.103772068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret") pod "global-pull-secret-syncer-8ld8d" (UID: "f044f5a4-1d02-48b3-a5c3-0dc9f2212f51") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:24.743136 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.743074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:24.743312 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.743222 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:32:24.743312 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.743253 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:32:24.743312 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.743266 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8mfr9 for pod openshift-network-diagnostics/network-check-target-q8p4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:24.743409 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:24.743323 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9 podName:8d47d830-d5d0-45ff-92d5-c605bd23d46f nodeName:}" failed. No retries permitted until 2026-09-10 15:32:25.743309017 +0000 UTC m=+4.305618185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mfr9" (UniqueName: "kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9") pod "network-check-target-q8p4v" (UID: "8d47d830-d5d0-45ff-92d5-c605bd23d46f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:24.966068 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:24.965791 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efed680_b002_4d32_8d04_33de60e395e4.slice/crio-e4cbe09766e1ef60fe92e2763878fdf66cd119ed6ed8b36e73bb754568347d37 WatchSource:0}: Error finding container e4cbe09766e1ef60fe92e2763878fdf66cd119ed6ed8b36e73bb754568347d37: Status 404 returned error can't find the container with id e4cbe09766e1ef60fe92e2763878fdf66cd119ed6ed8b36e73bb754568347d37 Sep 10 15:32:24.967273 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.967221 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:27:23 +0000 UTC" deadline="2028-02-19 07:24:23.132092726 +0000 UTC" Sep 10 15:32:24.967273 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:24.967272 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12639h51m58.164825567s" Sep 10 15:32:24.972982 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:24.972946 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bb48bf_062a_43c9_b426_ab56a1bb466f.slice/crio-c465347a2b19b1c646a2763e58b3b2752b0c88f5c9f30f81cf44afae0199c9f5 WatchSource:0}: Error finding container c465347a2b19b1c646a2763e58b3b2752b0c88f5c9f30f81cf44afae0199c9f5: Status 404 returned error can't find the container with id c465347a2b19b1c646a2763e58b3b2752b0c88f5c9f30f81cf44afae0199c9f5 Sep 10 15:32:24.974647 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:24.974334 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf636588a_67d1_4196_91d5_783f5ca11b8a.slice/crio-d40953c564624597fb8542cdbf23befab6197227690210c4ea8a8dfd46b7926c WatchSource:0}: Error finding container d40953c564624597fb8542cdbf23befab6197227690210c4ea8a8dfd46b7926c: Status 404 returned error can't find the container with id d40953c564624597fb8542cdbf23befab6197227690210c4ea8a8dfd46b7926c Sep 10 15:32:24.975665 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:24.975599 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed4497c_aa2c_4f87_a183_ac8d193d9e93.slice/crio-76683dd3fb41f480b3157ec61dc34f2d8485c480e01cbee1dff751d1d99fe039 WatchSource:0}: Error finding container 76683dd3fb41f480b3157ec61dc34f2d8485c480e01cbee1dff751d1d99fe039: Status 404 returned error can't find the container with id 76683dd3fb41f480b3157ec61dc34f2d8485c480e01cbee1dff751d1d99fe039 Sep 10 15:32:24.976919 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:24.976719 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92f14a2_c880_4843_98be_7a8480914160.slice/crio-347de32ce1ac437ff476d6559e18b1f3ab8f9acce640666f12595b7d42da20fa WatchSource:0}: Error finding container 347de32ce1ac437ff476d6559e18b1f3ab8f9acce640666f12595b7d42da20fa: Status 404 returned error can't find the container with id 347de32ce1ac437ff476d6559e18b1f3ab8f9acce640666f12595b7d42da20fa Sep 10 15:32:24.979084 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:24.978888 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf249678_090c_4a60_b691_634f0c9c131c.slice/crio-948582ec0b531b4aafa6378fad1d0be6573bc8a81eb9df882591ca85f1e02e54 WatchSource:0}: Error finding container 948582ec0b531b4aafa6378fad1d0be6573bc8a81eb9df882591ca85f1e02e54: Status 404 returned error can't find the container with id 948582ec0b531b4aafa6378fad1d0be6573bc8a81eb9df882591ca85f1e02e54 Sep 10 15:32:25.030258 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.030229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hvffx" event={"ID":"f636588a-67d1-4196-91d5-783f5ca11b8a","Type":"ContainerStarted","Data":"d40953c564624597fb8542cdbf23befab6197227690210c4ea8a8dfd46b7926c"} Sep 10 15:32:25.033476 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.033450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" event={"ID":"af249678-090c-4a60-b691-634f0c9c131c","Type":"ContainerStarted","Data":"948582ec0b531b4aafa6378fad1d0be6573bc8a81eb9df882591ca85f1e02e54"} Sep 10 15:32:25.040527 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.040504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rzbl8" event={"ID":"fed4497c-aa2c-4f87-a183-ac8d193d9e93","Type":"ContainerStarted","Data":"76683dd3fb41f480b3157ec61dc34f2d8485c480e01cbee1dff751d1d99fe039"} Sep 10 15:32:25.042916 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.042873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" event={"ID":"50bb48bf-062a-43c9-b426-ab56a1bb466f","Type":"ContainerStarted","Data":"c465347a2b19b1c646a2763e58b3b2752b0c88f5c9f30f81cf44afae0199c9f5"} Sep 10 15:32:25.043919 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.043888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hvnc" event={"ID":"b92f14a2-c880-4843-98be-7a8480914160","Type":"ContainerStarted","Data":"347de32ce1ac437ff476d6559e18b1f3ab8f9acce640666f12595b7d42da20fa"} Sep 10 15:32:25.044993 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.044959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rbdsp" event={"ID":"25d697d5-2458-4160-9e16-3fcef346035d","Type":"ContainerStarted","Data":"debd9e243c5f21dd583115ae8a0a2952a28696529b411db53fe18dea13bc5792"} Sep 10 15:32:25.046145 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.046121 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jh65b" event={"ID":"baa7bbbd-ba07-41eb-883f-4fa765f40091","Type":"ContainerStarted","Data":"3c7c693de13b3935a91cfbd6804f413daec412f05e10967078211c135d7b1344"} Sep 10 15:32:25.047269 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.047247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb459" event={"ID":"368dd85e-db5c-42da-b98c-b8cd54960d1b","Type":"ContainerStarted","Data":"19eac3a4f22090736b6ce1cb1b155d8309559776b90704cbbc2aba41632d3055"} Sep 10 15:32:25.048527 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.048503 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" event={"ID":"2efed680-b002-4d32-8d04-33de60e395e4","Type":"ContainerStarted","Data":"e4cbe09766e1ef60fe92e2763878fdf66cd119ed6ed8b36e73bb754568347d37"} Sep 10 15:32:25.550148 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.549333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:25.550148 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.549404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:25.550148 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:25.549572 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:25.550148 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:25.549635 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs podName:6150e082-626a-47a0-b8a0-5a9ea64b8251 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:27.549617573 +0000 UTC m=+6.111926755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs") pod "network-metrics-daemon-hvz5r" (UID: "6150e082-626a-47a0-b8a0-5a9ea64b8251") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:25.550148 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:25.549697 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:25.550148 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:25.549735 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret podName:f044f5a4-1d02-48b3-a5c3-0dc9f2212f51 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:27.549716488 +0000 UTC m=+6.112025663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret") pod "global-pull-secret-syncer-8ld8d" (UID: "f044f5a4-1d02-48b3-a5c3-0dc9f2212f51") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:25.751099 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.751055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:25.751271 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:25.751244 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:32:25.751271 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:25.751269 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:32:25.751395 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:25.751284 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8mfr9 for pod openshift-network-diagnostics/network-check-target-q8p4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:25.751395 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:25.751342 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9 podName:8d47d830-d5d0-45ff-92d5-c605bd23d46f nodeName:}" failed. No retries permitted until 2026-09-10 15:32:27.75132424 +0000 UTC m=+6.313633421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mfr9" (UniqueName: "kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9") pod "network-check-target-q8p4v" (UID: "8d47d830-d5d0-45ff-92d5-c605bd23d46f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:25.967669 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.967591 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:27:23 +0000 UTC" deadline="2028-02-25 16:35:07.24356295 +0000 UTC" Sep 10 15:32:25.967669 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:25.967634 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12793h2m41.275933846s" Sep 10 15:32:26.022578 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:26.022546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:26.022762 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:26.022584 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:26.022762 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:26.022676 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:26.023167 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:26.023128 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:26.023227 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:26.023191 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:26.023284 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:26.023260 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:26.095779 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:26.095735 2580 generic.go:356] "Generic (PLEG): container finished" podID="d99c6e4cbc478f4080fa6c464fe6df80" containerID="63faaf363a549a858c8ff7bba2ca8ae449348a5aa6131c7b2da86dfd7452a22a" exitCode=0 Sep 10 15:32:26.096187 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:26.096159 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" event={"ID":"d99c6e4cbc478f4080fa6c464fe6df80","Type":"ContainerDied","Data":"63faaf363a549a858c8ff7bba2ca8ae449348a5aa6131c7b2da86dfd7452a22a"} Sep 10 15:32:26.127959 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:26.127921 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" event={"ID":"3cdfc09abde163e5efcd6f1fa3f633fc","Type":"ContainerStarted","Data":"0235a350d5c0c6f9ea9827d87195d13226c9613bfa3262984c40cbabc111d1c7"} Sep 10 15:32:26.143275 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:26.142224 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-9.ec2.internal" podStartSLOduration=3.142203598 podStartE2EDuration="3.142203598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:26.141939026 +0000 UTC m=+4.704248217" watchObservedRunningTime="2026-09-10 15:32:26.142203598 +0000 UTC m=+4.704512792" Sep 10 15:32:27.145936 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:27.145818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" event={"ID":"d99c6e4cbc478f4080fa6c464fe6df80","Type":"ContainerStarted","Data":"d05d3191e73a1b9ff76ef198b61af025f8072ec90fffb8fd57bd16b3d22b59c6"} Sep 10 15:32:27.568934 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:27.568844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:27.569108 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:27.568940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:27.569108 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:27.569022 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:27.569108 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:27.569069 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:27.569108 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:27.569097 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs podName:6150e082-626a-47a0-b8a0-5a9ea64b8251 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:31.569077006 +0000 UTC m=+10.131386178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs") pod "network-metrics-daemon-hvz5r" (UID: "6150e082-626a-47a0-b8a0-5a9ea64b8251") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:27.569317 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:27.569122 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret podName:f044f5a4-1d02-48b3-a5c3-0dc9f2212f51 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:31.569106898 +0000 UTC m=+10.131416072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret") pod "global-pull-secret-syncer-8ld8d" (UID: "f044f5a4-1d02-48b3-a5c3-0dc9f2212f51") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:27.770673 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:27.770631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:27.770849 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:27.770828 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:32:27.770849 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:27.770847 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:32:27.770957 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:27.770859 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8mfr9 for pod openshift-network-diagnostics/network-check-target-q8p4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:27.770957 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:27.770918 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9 podName:8d47d830-d5d0-45ff-92d5-c605bd23d46f nodeName:}" failed. No retries permitted until 2026-09-10 15:32:31.770900162 +0000 UTC m=+10.333209344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mfr9" (UniqueName: "kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9") pod "network-check-target-q8p4v" (UID: "8d47d830-d5d0-45ff-92d5-c605bd23d46f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:28.022560 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:28.022524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:28.022560 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:28.022556 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:28.022814 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:28.022672 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:28.023268 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:28.023072 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:28.023268 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:28.023136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:28.023268 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:28.023211 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:30.023101 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:30.023058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:30.023707 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:30.023108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:30.023707 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:30.023058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:30.023707 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:30.023196 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:30.023707 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:30.023315 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:30.023707 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:30.023397 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:31.604533 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:31.604082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:31.604533 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:31.604159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:31.604533 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:31.604247 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:31.604533 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:31.604266 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:31.604533 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:31.604311 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret podName:f044f5a4-1d02-48b3-a5c3-0dc9f2212f51 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:39.604294264 +0000 UTC m=+18.166603448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret") pod "global-pull-secret-syncer-8ld8d" (UID: "f044f5a4-1d02-48b3-a5c3-0dc9f2212f51") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:31.604533 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:31.604328 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs podName:6150e082-626a-47a0-b8a0-5a9ea64b8251 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:39.604319752 +0000 UTC m=+18.166628919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs") pod "network-metrics-daemon-hvz5r" (UID: "6150e082-626a-47a0-b8a0-5a9ea64b8251") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:31.806635 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:31.806591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:31.806832 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:31.806803 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:32:31.806897 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:31.806836 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:32:31.806897 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:31.806851 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8mfr9 for pod openshift-network-diagnostics/network-check-target-q8p4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:31.806997 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:31.806914 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9 podName:8d47d830-d5d0-45ff-92d5-c605bd23d46f nodeName:}" failed. No retries permitted until 2026-09-10 15:32:39.806895266 +0000 UTC m=+18.369204447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mfr9" (UniqueName: "kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9") pod "network-check-target-q8p4v" (UID: "8d47d830-d5d0-45ff-92d5-c605bd23d46f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:32.023364 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:32.023322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:32.023552 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:32.023443 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:32.023552 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:32.023501 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:32.023552 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:32.023534 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:32.023670 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:32.023562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:32.023670 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:32.023629 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:34.022494 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:34.022402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:34.022945 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:34.022402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:34.022945 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:34.022575 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:34.022945 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:34.022640 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:34.022945 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:34.022402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:34.022945 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:34.022740 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:36.022913 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:36.022768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:36.022913 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:36.022768 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:36.022913 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:36.022820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:36.022913 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:36.022893 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:36.023529 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:36.022988 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:36.023529 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:36.023062 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:38.022806 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:38.022559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:38.023231 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:38.022581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:38.023231 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:38.022925 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:38.023231 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:38.022559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:38.023231 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:38.023051 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:38.023231 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:38.023046 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:39.666545 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:39.666504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:39.667029 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:39.666588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:39.667029 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:39.666684 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:39.667029 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:39.666762 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret podName:f044f5a4-1d02-48b3-a5c3-0dc9f2212f51 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:55.666743133 +0000 UTC m=+34.229052320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret") pod "global-pull-secret-syncer-8ld8d" (UID: "f044f5a4-1d02-48b3-a5c3-0dc9f2212f51") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:39.667029 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:39.666696 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:39.667029 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:39.666834 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs podName:6150e082-626a-47a0-b8a0-5a9ea64b8251 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:55.666816514 +0000 UTC m=+34.229125697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs") pod "network-metrics-daemon-hvz5r" (UID: "6150e082-626a-47a0-b8a0-5a9ea64b8251") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:39.868652 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:39.868602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:39.868825 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:39.868739 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:32:39.868825 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:39.868770 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:32:39.868825 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:39.868785 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8mfr9 for pod openshift-network-diagnostics/network-check-target-q8p4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:39.868947 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:39.868848 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9 podName:8d47d830-d5d0-45ff-92d5-c605bd23d46f nodeName:}" failed. No retries permitted until 2026-09-10 15:32:55.868834437 +0000 UTC m=+34.431143606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mfr9" (UniqueName: "kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9") pod "network-check-target-q8p4v" (UID: "8d47d830-d5d0-45ff-92d5-c605bd23d46f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:40.022734 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:40.022619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:40.022734 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:40.022659 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:40.022734 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:40.022626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:40.023003 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:40.022759 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:40.023003 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:40.022829 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:40.023003 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:40.022891 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:42.026066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:42.024958 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:42.026066 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:42.025118 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:42.026066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:42.025597 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:42.026066 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:42.025720 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:42.026066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:42.025775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:42.026066 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:42.025876 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:43.194588 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:43.194556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6hvnc" event={"ID":"b92f14a2-c880-4843-98be-7a8480914160","Type":"ContainerStarted","Data":"a70c1a543c1f9acf38a7fa41396fa853cec151b1a4ed010652dc7d121377440b"} Sep 10 15:32:43.203612 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:43.202659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rzbl8" event={"ID":"fed4497c-aa2c-4f87-a183-ac8d193d9e93","Type":"ContainerStarted","Data":"15504599559bf6f4b541a89884432c7ce8732b119d9f3096fe155479e581f2bb"} Sep 10 15:32:43.211745 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:43.211297 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-9.ec2.internal" podStartSLOduration=20.211277298 podStartE2EDuration="20.211277298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:27.16038644 +0000 UTC m=+5.722695631" watchObservedRunningTime="2026-09-10 15:32:43.211277298 +0000 UTC m=+21.773586489" Sep 10 15:32:43.229968 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:43.229111 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6hvnc" podStartSLOduration=3.253161053 podStartE2EDuration="21.229089739s" totalImagesPullingTime="17.975928686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:43.211801805 +0000 UTC m=+21.774110998" watchObservedRunningTime="2026-09-10 15:32:43.229089739 +0000 UTC m=+21.791398930" Sep 10 15:32:44.022754 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.022566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:44.022936 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.022560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:44.022936 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.022560 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:44.022936 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:44.022853 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:44.023114 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:44.022984 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:44.023114 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:44.023068 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:44.211696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.211662 2580 generic.go:356] "Generic (PLEG): container finished" podID="368dd85e-db5c-42da-b98c-b8cd54960d1b" containerID="18c2dd500a5b0548ab9cf43a102a34c682dc464372eed429fe1d29d6a0597e18" exitCode=0 Sep 10 15:32:44.212360 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.211753 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb459" event={"ID":"368dd85e-db5c-42da-b98c-b8cd54960d1b","Type":"ContainerDied","Data":"18c2dd500a5b0548ab9cf43a102a34c682dc464372eed429fe1d29d6a0597e18"} Sep 10 15:32:44.215183 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.215156 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" event={"ID":"2efed680-b002-4d32-8d04-33de60e395e4","Type":"ContainerStarted","Data":"98372da45ec92b0245fa398600175525464819880b170422463b5528d523a372"} Sep 10 15:32:44.215263 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.215250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" event={"ID":"2efed680-b002-4d32-8d04-33de60e395e4","Type":"ContainerStarted","Data":"2f3e164160e792683403fc50f8711e1c50f479229241caae4263808797f7e062"} Sep 10 15:32:44.215321 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.215308 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" event={"ID":"2efed680-b002-4d32-8d04-33de60e395e4","Type":"ContainerStarted","Data":"40db4197ebe20c42beefa9e12e3f18891e30e13b38d60fd31cba755f22dfb3d4"} Sep 10 15:32:44.215357 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.215345 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" event={"ID":"2efed680-b002-4d32-8d04-33de60e395e4","Type":"ContainerStarted","Data":"cba466ef13799e6a84f888df8ff855a3a7fced46da6f86c9f7d22bc2f9398b4f"} Sep 10 15:32:44.215404 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.215380 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" event={"ID":"2efed680-b002-4d32-8d04-33de60e395e4","Type":"ContainerStarted","Data":"fb931bf4a89009a07ff37fec41ce1a3086f8db52dea88b92cd9f218a2752a3ae"} Sep 10 15:32:44.215475 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.215434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" event={"ID":"2efed680-b002-4d32-8d04-33de60e395e4","Type":"ContainerStarted","Data":"39cd101082f5aab09bf9d87033591c70ebbb19a0c4a761a224f8bcda0c94d2bf"} Sep 10 15:32:44.216674 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.216649 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hvffx" event={"ID":"f636588a-67d1-4196-91d5-783f5ca11b8a","Type":"ContainerStarted","Data":"f6918ebbe61f16417a36c36fdeaa4a1d91828854982c2d43a499d49239819238"} Sep 10 15:32:44.218071 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.218050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" event={"ID":"af249678-090c-4a60-b691-634f0c9c131c","Type":"ContainerStarted","Data":"b07d6951677a792d89e2fb20830aed6349ba5d506e49c8b3255fde50bd85a487"} Sep 10 15:32:44.219514 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.219486 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" event={"ID":"50bb48bf-062a-43c9-b426-ab56a1bb466f","Type":"ContainerStarted","Data":"7747b2fa62f95c4377b85ae3a55828deb412ff4e8ddd2695d55c09d72644668f"} Sep 10 15:32:44.220887 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.220865 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jh65b" event={"ID":"baa7bbbd-ba07-41eb-883f-4fa765f40091","Type":"ContainerStarted","Data":"449e6bbe6e672a2b5cedba6ba4bddf8756e289adf07609cb04f7dbbd758dcd06"} Sep 10 15:32:44.230099 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.230061 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rzbl8" podStartSLOduration=8.229259505 podStartE2EDuration="22.230049421s" totalImagesPullingTime="14.000789916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:43.229800539 +0000 UTC m=+21.792109730" watchObservedRunningTime="2026-09-10 15:32:44.230049421 +0000 UTC m=+22.792358611" Sep 10 15:32:44.248847 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.248801 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mtjxt" podStartSLOduration=4.207463937 podStartE2EDuration="22.248787046s" totalImagesPullingTime="18.041323109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:44.248609085 +0000 UTC m=+22.810918275" watchObservedRunningTime="2026-09-10 15:32:44.248787046 +0000 UTC m=+22.811096235" Sep 10 15:32:44.262405 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.262353 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hvffx" podStartSLOduration=4.285892289 podStartE2EDuration="22.262338376s" totalImagesPullingTime="17.976446087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:44.261872601 +0000 UTC m=+22.824181791" watchObservedRunningTime="2026-09-10 15:32:44.262338376 +0000 UTC m=+22.824647565" Sep 10 15:32:44.277099 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.277016 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jh65b" podStartSLOduration=4.088335984 podStartE2EDuration="22.277002067s" totalImagesPullingTime="18.188666083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:44.276798493 +0000 UTC m=+22.839107683" watchObservedRunningTime="2026-09-10 15:32:44.277002067 +0000 UTC m=+22.839311256" Sep 10 15:32:44.523291 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:44.523256 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 15:32:45.045708 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:45.045567 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:32:44.52328557Z","UUID":"ebd42280-b7f6-4ce0-bd2e-11ccdabde208","Handler":null,"Name":"","Endpoint":""} Sep 10 15:32:45.047284 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:45.047258 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 15:32:45.047485 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:45.047298 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 15:32:45.225482 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:45.225442 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rbdsp" event={"ID":"25d697d5-2458-4160-9e16-3fcef346035d","Type":"ContainerStarted","Data":"d2b3ca512172af5743a57d241ca5cdccf6f2a46f537c0335a25f878f6b088d86"} Sep 10 15:32:45.228383 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:45.228323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" event={"ID":"af249678-090c-4a60-b691-634f0c9c131c","Type":"ContainerStarted","Data":"95becacd08e0e48239ba0a00f9723b629f8c932bf139189701386dd9acbb5346"} Sep 10 15:32:45.241695 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:45.241641 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rbdsp" podStartSLOduration=5.231416302 podStartE2EDuration="23.241621949s" totalImagesPullingTime="18.010205647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:45.241434301 +0000 UTC m=+23.803743487" watchObservedRunningTime="2026-09-10 15:32:45.241621949 +0000 UTC m=+23.803931139" Sep 10 15:32:46.022386 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:46.022074 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:46.022604 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:46.022414 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:46.022604 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:46.022102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:46.022604 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:46.022528 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:46.022604 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:46.022075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:46.022821 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:46.022616 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:46.235227 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:46.235101 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" event={"ID":"2efed680-b002-4d32-8d04-33de60e395e4","Type":"ContainerStarted","Data":"d9c08f5b917fabdc14716a1e9ded23ffa3231d44720a68a44185a4aa4cb1d634"} Sep 10 15:32:46.237658 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:46.237626 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" event={"ID":"af249678-090c-4a60-b691-634f0c9c131c","Type":"ContainerStarted","Data":"4d9fab8258d36eb92ed9e4fc0c16f6d80e1b2a3b1ba0f5d57a02c788a839cc88"} Sep 10 15:32:46.256597 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:46.256545 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mgl2b" podStartSLOduration=4.020159613 podStartE2EDuration="24.256527106s" totalImagesPullingTime="20.236367493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:46.256208822 +0000 UTC m=+24.818518032" watchObservedRunningTime="2026-09-10 15:32:46.256527106 +0000 UTC m=+24.818836314" Sep 10 15:32:47.202342 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:47.202293 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:47.202940 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:47.202916 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:47.240783 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:47.240711 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:47.241223 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:47.241202 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rzbl8" Sep 10 15:32:48.022484 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:48.022415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:48.022666 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:48.022581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:48.022666 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:48.022586 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:48.022784 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:48.022700 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:48.022784 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:48.022775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:48.022906 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:48.022868 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:49.246959 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:49.246790 2580 generic.go:356] "Generic (PLEG): container finished" podID="368dd85e-db5c-42da-b98c-b8cd54960d1b" containerID="22a6532405ae937195ce2cb872f4afb28081147b876a9442e9a765e965f2836b" exitCode=0 Sep 10 15:32:49.247609 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:49.246877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb459" event={"ID":"368dd85e-db5c-42da-b98c-b8cd54960d1b","Type":"ContainerDied","Data":"22a6532405ae937195ce2cb872f4afb28081147b876a9442e9a765e965f2836b"} Sep 10 15:32:49.251063 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:49.251038 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" event={"ID":"2efed680-b002-4d32-8d04-33de60e395e4","Type":"ContainerStarted","Data":"871a34f4c50b0551c6a0621a54ea618141d41b12eceabd2eb7c398942e5ab64c"} Sep 10 15:32:49.295496 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:49.295441 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" podStartSLOduration=9.243483383 podStartE2EDuration="27.295408735s" totalImagesPullingTime="18.051925352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:49.295350607 +0000 UTC m=+27.857659890" watchObservedRunningTime="2026-09-10 15:32:49.295408735 +0000 UTC m=+27.857717921" Sep 10 15:32:50.022461 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.022414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:50.022677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.022414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:50.022677 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:50.022541 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:50.022677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.022414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:50.022677 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:50.022625 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:50.022907 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:50.022784 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:50.206063 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.206030 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8ld8d"] Sep 10 15:32:50.208636 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.208536 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8p4v"] Sep 10 15:32:50.208636 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.208592 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hvz5r"] Sep 10 15:32:50.252871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.252849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:50.252871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.252857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:50.253531 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.252849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:50.253531 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:50.252949 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:50.253531 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:50.253030 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:50.253531 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.253118 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:50.253531 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:50.253490 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:50.253531 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.253523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:50.253722 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.253572 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:50.267941 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.267910 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:50.268071 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:50.268012 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:32:51.257261 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:51.257017 2580 generic.go:356] "Generic (PLEG): container finished" podID="368dd85e-db5c-42da-b98c-b8cd54960d1b" containerID="59c2feab0e11e238370d001482996a02ce5dbfac89616b763ee7de0062730732" exitCode=0 Sep 10 15:32:51.257644 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:51.257077 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb459" event={"ID":"368dd85e-db5c-42da-b98c-b8cd54960d1b","Type":"ContainerDied","Data":"59c2feab0e11e238370d001482996a02ce5dbfac89616b763ee7de0062730732"} Sep 10 15:32:52.023853 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:52.023823 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:52.023980 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:52.023908 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:52.023980 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:52.023942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:52.024176 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:52.023963 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:52.024176 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:52.024018 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:52.024176 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:52.024112 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:52.263115 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:52.263075 2580 generic.go:356] "Generic (PLEG): container finished" podID="368dd85e-db5c-42da-b98c-b8cd54960d1b" containerID="6d4e5c6a775243bd9bfb073cbfdb5ae7acad79bd7571f94a654d97d4072add18" exitCode=0 Sep 10 15:32:52.263491 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:52.263170 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb459" event={"ID":"368dd85e-db5c-42da-b98c-b8cd54960d1b","Type":"ContainerDied","Data":"6d4e5c6a775243bd9bfb073cbfdb5ae7acad79bd7571f94a654d97d4072add18"} Sep 10 15:32:54.022476 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:54.022439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:54.022476 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:54.022470 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:54.023171 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:54.022439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:54.023171 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:54.022593 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hvz5r" podUID="6150e082-626a-47a0-b8a0-5a9ea64b8251" Sep 10 15:32:54.023171 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:54.022644 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8ld8d" podUID="f044f5a4-1d02-48b3-a5c3-0dc9f2212f51" Sep 10 15:32:54.023171 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:54.022724 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q8p4v" podUID="8d47d830-d5d0-45ff-92d5-c605bd23d46f" Sep 10 15:32:55.656468 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.656415 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-9.ec2.internal" event="NodeReady" Sep 10 15:32:55.656941 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.656596 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 15:32:55.684937 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.684895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:55.685113 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.684967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:55.685113 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.685050 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:55.685205 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.685119 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs podName:6150e082-626a-47a0-b8a0-5a9ea64b8251 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:27.685097342 +0000 UTC m=+66.247406511 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs") pod "network-metrics-daemon-hvz5r" (UID: "6150e082-626a-47a0-b8a0-5a9ea64b8251") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:32:55.685205 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.685050 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:55.685295 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.685218 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret podName:f044f5a4-1d02-48b3-a5c3-0dc9f2212f51 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:27.685198284 +0000 UTC m=+66.247507452 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret") pod "global-pull-secret-syncer-8ld8d" (UID: "f044f5a4-1d02-48b3-a5c3-0dc9f2212f51") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:32:55.700058 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.700032 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m5dct"] Sep 10 15:32:55.712754 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.712722 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-htnn6"] Sep 10 15:32:55.712951 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.712930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.715103 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.715078 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 15:32:55.715253 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.715104 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 15:32:55.715466 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.715445 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bddbj\"" Sep 10 15:32:55.724907 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.724876 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m5dct"] Sep 10 15:32:55.725060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.724924 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-htnn6"] Sep 10 15:32:55.725060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.724995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-htnn6" Sep 10 15:32:55.727322 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.727298 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 15:32:55.727482 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.727461 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t4nnn\"" Sep 10 15:32:55.728926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.728242 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 15:32:55.728926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.728271 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 15:32:55.785932 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.785884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.785932 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.785934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gr2\" (UniqueName: \"kubernetes.io/projected/9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd-kube-api-access-49gr2\") pod \"ingress-canary-htnn6\" (UID: \"9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd\") " pod="openshift-ingress-canary/ingress-canary-htnn6" Sep 10 15:32:55.786197 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.785982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg4j5\" (UniqueName: \"kubernetes.io/projected/fa218d1e-4464-49e6-b1e8-141f0dd3aace-kube-api-access-kg4j5\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.786197 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.786050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd-cert\") pod \"ingress-canary-htnn6\" (UID: \"9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd\") " pod="openshift-ingress-canary/ingress-canary-htnn6" Sep 10 15:32:55.786197 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.786139 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa218d1e-4464-49e6-b1e8-141f0dd3aace-tmp-dir\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.786342 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.786202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa218d1e-4464-49e6-b1e8-141f0dd3aace-config-volume\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.887054 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.886953 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa218d1e-4464-49e6-b1e8-141f0dd3aace-tmp-dir\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.887291 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.887120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa218d1e-4464-49e6-b1e8-141f0dd3aace-config-volume\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.887291 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.887153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:55.887291 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.887180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.887291 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.887207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gr2\" (UniqueName: \"kubernetes.io/projected/9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd-kube-api-access-49gr2\") pod \"ingress-canary-htnn6\" (UID: \"9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd\") " pod="openshift-ingress-canary/ingress-canary-htnn6" Sep 10 15:32:55.887291 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.887265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg4j5\" (UniqueName: \"kubernetes.io/projected/fa218d1e-4464-49e6-b1e8-141f0dd3aace-kube-api-access-kg4j5\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.887291 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.887276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa218d1e-4464-49e6-b1e8-141f0dd3aace-tmp-dir\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.887609 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.887294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd-cert\") pod \"ingress-canary-htnn6\" (UID: \"9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd\") " pod="openshift-ingress-canary/ingress-canary-htnn6" Sep 10 15:32:55.887609 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.887325 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:32:55.887609 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.887335 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:32:55.887609 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.887417 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls podName:fa218d1e-4464-49e6-b1e8-141f0dd3aace nodeName:}" failed. No retries permitted until 2026-09-10 15:32:56.387394386 +0000 UTC m=+34.949703571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls") pod "dns-default-m5dct" (UID: "fa218d1e-4464-49e6-b1e8-141f0dd3aace") : secret "dns-default-metrics-tls" not found Sep 10 15:32:55.887609 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.887351 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:32:55.887609 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.887454 2580 projected.go:196] Error preparing data for projected volume kube-api-access-8mfr9 for pod openshift-network-diagnostics/network-check-target-q8p4v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:55.887609 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:55.887496 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9 podName:8d47d830-d5d0-45ff-92d5-c605bd23d46f nodeName:}" failed. No retries permitted until 2026-09-10 15:33:27.887483256 +0000 UTC m=+66.449792437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mfr9" (UniqueName: "kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9") pod "network-check-target-q8p4v" (UID: "8d47d830-d5d0-45ff-92d5-c605bd23d46f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:32:55.887906 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.887758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa218d1e-4464-49e6-b1e8-141f0dd3aace-config-volume\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.892192 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.892163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd-cert\") pod \"ingress-canary-htnn6\" (UID: \"9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd\") " pod="openshift-ingress-canary/ingress-canary-htnn6" Sep 10 15:32:55.897208 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.897185 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg4j5\" (UniqueName: \"kubernetes.io/projected/fa218d1e-4464-49e6-b1e8-141f0dd3aace-kube-api-access-kg4j5\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:55.897366 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:55.897343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gr2\" (UniqueName: \"kubernetes.io/projected/9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd-kube-api-access-49gr2\") pod \"ingress-canary-htnn6\" (UID: \"9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd\") " pod="openshift-ingress-canary/ingress-canary-htnn6" Sep 10 15:32:56.022558 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.022526 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:32:56.022754 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.022526 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:32:56.022754 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.022526 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:32:56.025760 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.025535 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 15:32:56.025760 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.025560 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 15:32:56.025760 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.025576 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 15:32:56.025760 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.025535 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gcptc\"" Sep 10 15:32:56.025760 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.025535 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-skj7z\"" Sep 10 15:32:56.026056 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.025953 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 15:32:56.036913 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.036891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-htnn6" Sep 10 15:32:56.391952 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.391911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:56.392152 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:56.392120 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:32:56.392221 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:56.392205 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls podName:fa218d1e-4464-49e6-b1e8-141f0dd3aace nodeName:}" failed. No retries permitted until 2026-09-10 15:32:57.392178655 +0000 UTC m=+35.954487824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls") pod "dns-default-m5dct" (UID: "fa218d1e-4464-49e6-b1e8-141f0dd3aace") : secret "dns-default-metrics-tls" not found Sep 10 15:32:56.954991 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:56.954780 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-htnn6"] Sep 10 15:32:56.958891 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:32:56.958868 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0fc599_85ef_4ad2_a3d6_ed614c8c6fbd.slice/crio-bb1823a9536a4dc36ac615d449600ca7e81375074883e5b87572157faff7d897 WatchSource:0}: Error finding container bb1823a9536a4dc36ac615d449600ca7e81375074883e5b87572157faff7d897: Status 404 returned error can't find the container with id bb1823a9536a4dc36ac615d449600ca7e81375074883e5b87572157faff7d897 Sep 10 15:32:57.279326 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:57.279231 2580 generic.go:356] "Generic (PLEG): container finished" podID="368dd85e-db5c-42da-b98c-b8cd54960d1b" containerID="b28c96af92953ce322273560242ad9f1d6fe03ff1cc33a6b39a0725ee6b84b4e" exitCode=0 Sep 10 15:32:57.279326 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:57.279314 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb459" event={"ID":"368dd85e-db5c-42da-b98c-b8cd54960d1b","Type":"ContainerDied","Data":"b28c96af92953ce322273560242ad9f1d6fe03ff1cc33a6b39a0725ee6b84b4e"} Sep 10 15:32:57.280413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:57.280388 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-htnn6" event={"ID":"9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd","Type":"ContainerStarted","Data":"bb1823a9536a4dc36ac615d449600ca7e81375074883e5b87572157faff7d897"} Sep 10 15:32:57.401121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:57.401078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:57.401276 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:57.401261 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:32:57.401343 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:57.401334 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls podName:fa218d1e-4464-49e6-b1e8-141f0dd3aace nodeName:}" failed. No retries permitted until 2026-09-10 15:32:59.401313549 +0000 UTC m=+37.963622720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls") pod "dns-default-m5dct" (UID: "fa218d1e-4464-49e6-b1e8-141f0dd3aace") : secret "dns-default-metrics-tls" not found Sep 10 15:32:58.287105 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:58.287016 2580 generic.go:356] "Generic (PLEG): container finished" podID="368dd85e-db5c-42da-b98c-b8cd54960d1b" containerID="5dc7ab9285c2694a0b9326f543c08adcecd5cc272088b2f91e920f895898c40b" exitCode=0 Sep 10 15:32:58.287105 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:58.287065 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb459" event={"ID":"368dd85e-db5c-42da-b98c-b8cd54960d1b","Type":"ContainerDied","Data":"5dc7ab9285c2694a0b9326f543c08adcecd5cc272088b2f91e920f895898c40b"} Sep 10 15:32:59.293137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:59.293103 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xb459" event={"ID":"368dd85e-db5c-42da-b98c-b8cd54960d1b","Type":"ContainerStarted","Data":"9c502371e6c9376b56ab0709af448dbcc0a2fec9c85f586c47b1080f8dd6f104"} Sep 10 15:32:59.315998 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:59.315944 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xb459" podStartSLOduration=8.952279621 podStartE2EDuration="37.315927596s" totalImagesPullingTime="28.363647975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:59.314364451 +0000 UTC m=+37.876673641" watchObservedRunningTime="2026-09-10 15:32:59.315927596 +0000 UTC m=+37.878236785" Sep 10 15:32:59.418685 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:32:59.418651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:32:59.418882 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:59.418831 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:32:59.418938 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:32:59.418895 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls podName:fa218d1e-4464-49e6-b1e8-141f0dd3aace nodeName:}" failed. No retries permitted until 2026-09-10 15:33:03.418879102 +0000 UTC m=+41.981188269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls") pod "dns-default-m5dct" (UID: "fa218d1e-4464-49e6-b1e8-141f0dd3aace") : secret "dns-default-metrics-tls" not found Sep 10 15:33:00.296939 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:00.296713 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-htnn6" event={"ID":"9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd","Type":"ContainerStarted","Data":"55507fee394f577d44378188e702e89e4894eaa9ec45ff2e05fb3e225f3b7536"} Sep 10 15:33:00.311343 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:00.311287 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-htnn6" podStartSLOduration=2.740175155 podStartE2EDuration="5.311268711s" totalImagesPullingTime="2.571093556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:00.310704027 +0000 UTC m=+38.873013218" watchObservedRunningTime="2026-09-10 15:33:00.311268711 +0000 UTC m=+38.873577901" Sep 10 15:33:03.449650 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:03.449593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:33:03.450026 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:03.449749 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:33:03.450026 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:03.449814 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls podName:fa218d1e-4464-49e6-b1e8-141f0dd3aace nodeName:}" failed. No retries permitted until 2026-09-10 15:33:11.449797824 +0000 UTC m=+50.012106992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls") pod "dns-default-m5dct" (UID: "fa218d1e-4464-49e6-b1e8-141f0dd3aace") : secret "dns-default-metrics-tls" not found Sep 10 15:33:05.594745 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.594705 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-sz8sl"] Sep 10 15:33:05.620023 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.619994 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-sz8sl"] Sep 10 15:33:05.620191 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.620034 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz"] Sep 10 15:33:05.620191 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.620141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-sz8sl" Sep 10 15:33:05.622518 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.622497 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jcs7h\"" Sep 10 15:33:05.635673 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.635647 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n"] Sep 10 15:33:05.635813 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.635793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz" Sep 10 15:33:05.638167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.638145 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 15:33:05.638285 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.638236 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kwgf6\"" Sep 10 15:33:05.638285 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.638262 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:33:05.653843 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.653818 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz"] Sep 10 15:33:05.653967 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.653858 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n"] Sep 10 15:33:05.653967 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.653918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.656081 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.656061 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:33:05.656193 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.656153 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-26h8s\"" Sep 10 15:33:05.656250 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.656225 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 15:33:05.656299 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.656283 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 15:33:05.656437 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.656396 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 15:33:05.697404 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.697378 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d6966675f-zbjnc"] Sep 10 15:33:05.711720 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.711695 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zkn28"] Sep 10 15:33:05.711837 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.711772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.714682 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.714665 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 15:33:05.714795 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.714777 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wjsnb\"" Sep 10 15:33:05.715164 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.715147 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 15:33:05.716525 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.716509 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 15:33:05.721320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.721280 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 15:33:05.725850 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.725830 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d6966675f-zbjnc"] Sep 10 15:33:05.725929 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.725867 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zkn28"] Sep 10 15:33:05.725984 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.725960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.728255 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.728236 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 10 15:33:05.728348 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.728262 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 10 15:33:05.728348 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.728279 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 10 15:33:05.728348 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.728238 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ltqkn\"" Sep 10 15:33:05.728348 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.728262 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:33:05.735257 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.735240 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 10 15:33:05.765284 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.765251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htbk\" (UniqueName: \"kubernetes.io/projected/50a8ad50-e704-43b5-ac07-734b0a43ce0b-kube-api-access-2htbk\") pod \"service-ca-operator-859f94c4b-drb9n\" (UID: \"50a8ad50-e704-43b5-ac07-734b0a43ce0b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.765413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.765289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p659n\" (UniqueName: \"kubernetes.io/projected/e45a16ee-815d-4e40-b619-76a671086c16-kube-api-access-p659n\") pod \"network-check-source-685456655f-sz8sl\" (UID: \"e45a16ee-815d-4e40-b619-76a671086c16\") " pod="openshift-network-diagnostics/network-check-source-685456655f-sz8sl" Sep 10 15:33:05.765413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.765389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50a8ad50-e704-43b5-ac07-734b0a43ce0b-serving-cert\") pod \"service-ca-operator-859f94c4b-drb9n\" (UID: \"50a8ad50-e704-43b5-ac07-734b0a43ce0b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.765534 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.765418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50a8ad50-e704-43b5-ac07-734b0a43ce0b-config\") pod \"service-ca-operator-859f94c4b-drb9n\" (UID: \"50a8ad50-e704-43b5-ac07-734b0a43ce0b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.765534 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.765460 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nr87\" (UniqueName: \"kubernetes.io/projected/ab259f57-795d-463b-b058-ccfe8d326b3d-kube-api-access-7nr87\") pod \"volume-data-source-validator-695c895bd6-cvzzz\" (UID: \"ab259f57-795d-463b-b058-ccfe8d326b3d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz" Sep 10 15:33:05.866486 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50a8ad50-e704-43b5-ac07-734b0a43ce0b-serving-cert\") pod \"service-ca-operator-859f94c4b-drb9n\" (UID: \"50a8ad50-e704-43b5-ac07-734b0a43ce0b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.866643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50a8ad50-e704-43b5-ac07-734b0a43ce0b-config\") pod \"service-ca-operator-859f94c4b-drb9n\" (UID: \"50a8ad50-e704-43b5-ac07-734b0a43ce0b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.866643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-image-registry-private-configuration\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.866643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-trusted-ca\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.866643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nr87\" (UniqueName: \"kubernetes.io/projected/ab259f57-795d-463b-b058-ccfe8d326b3d-kube-api-access-7nr87\") pod \"volume-data-source-validator-695c895bd6-cvzzz\" (UID: \"ab259f57-795d-463b-b058-ccfe8d326b3d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz" Sep 10 15:33:05.866643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecc661e5-071a-44c4-a08d-7cf930601269-serving-cert\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.866643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-certificates\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.866643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-bound-sa-token\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.866966 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ss5d\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-kube-api-access-6ss5d\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.866966 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecc661e5-071a-44c4-a08d-7cf930601269-config\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.866966 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htbk\" (UniqueName: \"kubernetes.io/projected/50a8ad50-e704-43b5-ac07-734b0a43ce0b-kube-api-access-2htbk\") pod \"service-ca-operator-859f94c4b-drb9n\" (UID: \"50a8ad50-e704-43b5-ac07-734b0a43ce0b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.866966 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.866856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc661e5-071a-44c4-a08d-7cf930601269-trusted-ca\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.867490 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.867462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-ca-trust-extracted\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.867605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.867508 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-installation-pull-secrets\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.867605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.867545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p659n\" (UniqueName: \"kubernetes.io/projected/e45a16ee-815d-4e40-b619-76a671086c16-kube-api-access-p659n\") pod \"network-check-source-685456655f-sz8sl\" (UID: \"e45a16ee-815d-4e40-b619-76a671086c16\") " pod="openshift-network-diagnostics/network-check-source-685456655f-sz8sl" Sep 10 15:33:05.867605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.867589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.867752 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.867625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlb6c\" (UniqueName: \"kubernetes.io/projected/ecc661e5-071a-44c4-a08d-7cf930601269-kube-api-access-qlb6c\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.867752 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.867680 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/50a8ad50-e704-43b5-ac07-734b0a43ce0b-config\") pod \"service-ca-operator-859f94c4b-drb9n\" (UID: \"50a8ad50-e704-43b5-ac07-734b0a43ce0b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.871665 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.871631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50a8ad50-e704-43b5-ac07-734b0a43ce0b-serving-cert\") pod \"service-ca-operator-859f94c4b-drb9n\" (UID: \"50a8ad50-e704-43b5-ac07-734b0a43ce0b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.876074 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.876045 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htbk\" (UniqueName: \"kubernetes.io/projected/50a8ad50-e704-43b5-ac07-734b0a43ce0b-kube-api-access-2htbk\") pod \"service-ca-operator-859f94c4b-drb9n\" (UID: \"50a8ad50-e704-43b5-ac07-734b0a43ce0b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.876407 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.876387 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p659n\" (UniqueName: \"kubernetes.io/projected/e45a16ee-815d-4e40-b619-76a671086c16-kube-api-access-p659n\") pod \"network-check-source-685456655f-sz8sl\" (UID: \"e45a16ee-815d-4e40-b619-76a671086c16\") " pod="openshift-network-diagnostics/network-check-source-685456655f-sz8sl" Sep 10 15:33:05.876488 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.876465 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nr87\" (UniqueName: \"kubernetes.io/projected/ab259f57-795d-463b-b058-ccfe8d326b3d-kube-api-access-7nr87\") pod \"volume-data-source-validator-695c895bd6-cvzzz\" (UID: \"ab259f57-795d-463b-b058-ccfe8d326b3d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz" Sep 10 15:33:05.929576 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.929541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-sz8sl" Sep 10 15:33:05.943964 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.943938 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz" Sep 10 15:33:05.961567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.961545 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" Sep 10 15:33:05.968822 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.968800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-image-registry-private-configuration\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.968903 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.968833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-trusted-ca\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.968903 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.968855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecc661e5-071a-44c4-a08d-7cf930601269-serving-cert\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.968903 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.968873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-certificates\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.969054 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.969003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-bound-sa-token\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.969054 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.969038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ss5d\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-kube-api-access-6ss5d\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.969159 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.969066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecc661e5-071a-44c4-a08d-7cf930601269-config\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.969159 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.969109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc661e5-071a-44c4-a08d-7cf930601269-trusted-ca\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.969159 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.969137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-ca-trust-extracted\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.969307 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.969160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-installation-pull-secrets\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.969307 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.969192 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.969307 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.969222 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlb6c\" (UniqueName: \"kubernetes.io/projected/ecc661e5-071a-44c4-a08d-7cf930601269-kube-api-access-qlb6c\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.970714 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:05.969640 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:33:05.970714 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:05.969662 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6966675f-zbjnc: secret "image-registry-tls" not found Sep 10 15:33:05.970714 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:05.969728 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls podName:d1d7ffd5-c3f4-4e66-a1be-aa7be0310165 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:06.469709596 +0000 UTC m=+45.032018765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls") pod "image-registry-6d6966675f-zbjnc" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165") : secret "image-registry-tls" not found Sep 10 15:33:05.970714 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.970075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-trusted-ca\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.970714 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.970187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecc661e5-071a-44c4-a08d-7cf930601269-config\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.970714 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.970523 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-ca-trust-extracted\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.970714 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.970701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-certificates\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.971118 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.970780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc661e5-071a-44c4-a08d-7cf930601269-trusted-ca\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.971702 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.971667 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-image-registry-private-configuration\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.971952 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.971933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecc661e5-071a-44c4-a08d-7cf930601269-serving-cert\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.972082 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.972057 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-installation-pull-secrets\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.977955 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.977934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlb6c\" (UniqueName: \"kubernetes.io/projected/ecc661e5-071a-44c4-a08d-7cf930601269-kube-api-access-qlb6c\") pod \"console-operator-85b454874b-zkn28\" (UID: \"ecc661e5-071a-44c4-a08d-7cf930601269\") " pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:05.978152 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.978128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-bound-sa-token\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:05.978580 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:05.978562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ss5d\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-kube-api-access-6ss5d\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:06.034838 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:06.034807 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:06.230069 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:06.229983 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-sz8sl"] Sep 10 15:33:06.233306 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:06.233276 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45a16ee_815d_4e40_b619_76a671086c16.slice/crio-d4cbf744dd1cd039158323b213b7d1281ca8e0f14433ef622adc2bb9232fadea WatchSource:0}: Error finding container d4cbf744dd1cd039158323b213b7d1281ca8e0f14433ef622adc2bb9232fadea: Status 404 returned error can't find the container with id d4cbf744dd1cd039158323b213b7d1281ca8e0f14433ef622adc2bb9232fadea Sep 10 15:33:06.248953 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:06.248928 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz"] Sep 10 15:33:06.252006 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:06.251984 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab259f57_795d_463b_b058_ccfe8d326b3d.slice/crio-14526b8fbc2b7d9537da3fe81456146cc5079872df56fa33e75d3f2f57feaef9 WatchSource:0}: Error finding container 14526b8fbc2b7d9537da3fe81456146cc5079872df56fa33e75d3f2f57feaef9: Status 404 returned error can't find the container with id 14526b8fbc2b7d9537da3fe81456146cc5079872df56fa33e75d3f2f57feaef9 Sep 10 15:33:06.283165 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:06.283139 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n"] Sep 10 15:33:06.286085 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:06.286058 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a8ad50_e704_43b5_ac07_734b0a43ce0b.slice/crio-460a1f320102640533cfdffc8144132af30762633096404ab5a389c4fa8e03af WatchSource:0}: Error finding container 460a1f320102640533cfdffc8144132af30762633096404ab5a389c4fa8e03af: Status 404 returned error can't find the container with id 460a1f320102640533cfdffc8144132af30762633096404ab5a389c4fa8e03af Sep 10 15:33:06.314913 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:06.314885 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-sz8sl" event={"ID":"e45a16ee-815d-4e40-b619-76a671086c16","Type":"ContainerStarted","Data":"d4cbf744dd1cd039158323b213b7d1281ca8e0f14433ef622adc2bb9232fadea"} Sep 10 15:33:06.315951 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:06.315932 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" event={"ID":"50a8ad50-e704-43b5-ac07-734b0a43ce0b","Type":"ContainerStarted","Data":"460a1f320102640533cfdffc8144132af30762633096404ab5a389c4fa8e03af"} Sep 10 15:33:06.316995 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:06.316975 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz" event={"ID":"ab259f57-795d-463b-b058-ccfe8d326b3d","Type":"ContainerStarted","Data":"14526b8fbc2b7d9537da3fe81456146cc5079872df56fa33e75d3f2f57feaef9"} Sep 10 15:33:06.329900 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:06.329870 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-zkn28"] Sep 10 15:33:06.332468 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:06.332447 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc661e5_071a_44c4_a08d_7cf930601269.slice/crio-1585f44e213fab91fec2aa92e96983f12729434a781fc8799be81d0ffc4212fa WatchSource:0}: Error finding container 1585f44e213fab91fec2aa92e96983f12729434a781fc8799be81d0ffc4212fa: Status 404 returned error can't find the container with id 1585f44e213fab91fec2aa92e96983f12729434a781fc8799be81d0ffc4212fa Sep 10 15:33:06.473497 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:06.473462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:06.473660 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:06.473544 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:33:06.473660 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:06.473565 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6966675f-zbjnc: secret "image-registry-tls" not found Sep 10 15:33:06.473660 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:06.473629 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls podName:d1d7ffd5-c3f4-4e66-a1be-aa7be0310165 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:07.473611484 +0000 UTC m=+46.035920651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls") pod "image-registry-6d6966675f-zbjnc" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165") : secret "image-registry-tls" not found Sep 10 15:33:07.322448 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:07.322381 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zkn28" event={"ID":"ecc661e5-071a-44c4-a08d-7cf930601269","Type":"ContainerStarted","Data":"1585f44e213fab91fec2aa92e96983f12729434a781fc8799be81d0ffc4212fa"} Sep 10 15:33:07.482273 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:07.482228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:07.482571 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:07.482374 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:33:07.482571 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:07.482392 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6966675f-zbjnc: secret "image-registry-tls" not found Sep 10 15:33:07.482571 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:07.482480 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls podName:d1d7ffd5-c3f4-4e66-a1be-aa7be0310165 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:09.482457639 +0000 UTC m=+48.044766807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls") pod "image-registry-6d6966675f-zbjnc" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165") : secret "image-registry-tls" not found Sep 10 15:33:09.038868 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.038790 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-5m5tq"] Sep 10 15:33:09.057564 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.057527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-5m5tq"] Sep 10 15:33:09.057725 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.057610 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:09.059956 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.059882 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 15:33:09.059956 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.059952 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9cpbh\"" Sep 10 15:33:09.060164 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.059974 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 15:33:09.196677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.196640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/909ac02b-6554-404c-8b05-572d1d5902a9-nginx-conf\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:09.196841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.196685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:09.297495 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.297397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/909ac02b-6554-404c-8b05-572d1d5902a9-nginx-conf\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:09.297495 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.297470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:09.297710 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:09.297619 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 15:33:09.297710 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:09.297687 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert podName:909ac02b-6554-404c-8b05-572d1d5902a9 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:09.797671597 +0000 UTC m=+48.359980765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5m5tq" (UID: "909ac02b-6554-404c-8b05-572d1d5902a9") : secret "networking-console-plugin-cert" not found Sep 10 15:33:09.298127 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.298103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/909ac02b-6554-404c-8b05-572d1d5902a9-nginx-conf\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:09.499654 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.499603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:09.499838 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:09.499813 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:33:09.499880 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:09.499843 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6966675f-zbjnc: secret "image-registry-tls" not found Sep 10 15:33:09.499935 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:09.499917 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls podName:d1d7ffd5-c3f4-4e66-a1be-aa7be0310165 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:13.49989427 +0000 UTC m=+52.062203443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls") pod "image-registry-6d6966675f-zbjnc" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165") : secret "image-registry-tls" not found Sep 10 15:33:09.802572 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.802524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:09.802775 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:09.802708 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 15:33:09.802838 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:09.802795 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert podName:909ac02b-6554-404c-8b05-572d1d5902a9 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:10.802772566 +0000 UTC m=+49.365081740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5m5tq" (UID: "909ac02b-6554-404c-8b05-572d1d5902a9") : secret "networking-console-plugin-cert" not found Sep 10 15:33:09.809878 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.809853 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w8np8"] Sep 10 15:33:09.843292 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.843257 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w8np8"] Sep 10 15:33:09.843488 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.843402 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:09.846373 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.846349 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 15:33:09.846508 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.846392 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 15:33:09.846508 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.846442 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 15:33:09.846700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.846676 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cj5pp\"" Sep 10 15:33:09.846824 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.846682 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 15:33:09.858066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.858034 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph"] Sep 10 15:33:09.879034 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.879008 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph"] Sep 10 15:33:09.879178 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.879135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph" Sep 10 15:33:09.881646 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.881621 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gp92x\"" Sep 10 15:33:09.881798 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.881710 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 15:33:09.881798 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:09.881754 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 15:33:10.004600 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.004562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfz4b\" (UniqueName: \"kubernetes.io/projected/325fcde8-67ae-4bfa-97bd-f6166cc1471d-kube-api-access-bfz4b\") pod \"migrator-fdbb8fdff-wv5ph\" (UID: \"325fcde8-67ae-4bfa-97bd-f6166cc1471d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph" Sep 10 15:33:10.004787 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.004624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.004787 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.004667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/028c54f4-e5fc-4f58-9127-390792d644ef-crio-socket\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.004787 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.004708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2snn\" (UniqueName: \"kubernetes.io/projected/028c54f4-e5fc-4f58-9127-390792d644ef-kube-api-access-g2snn\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.004787 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.004755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/028c54f4-e5fc-4f58-9127-390792d644ef-data-volume\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.005007 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.004873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.106003 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.105916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/028c54f4-e5fc-4f58-9127-390792d644ef-crio-socket\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.106003 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.105991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2snn\" (UniqueName: \"kubernetes.io/projected/028c54f4-e5fc-4f58-9127-390792d644ef-kube-api-access-g2snn\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.106571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.106031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/028c54f4-e5fc-4f58-9127-390792d644ef-data-volume\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.106571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.106115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.106571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.106165 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/028c54f4-e5fc-4f58-9127-390792d644ef-crio-socket\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.106571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.106174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfz4b\" (UniqueName: \"kubernetes.io/projected/325fcde8-67ae-4bfa-97bd-f6166cc1471d-kube-api-access-bfz4b\") pod \"migrator-fdbb8fdff-wv5ph\" (UID: \"325fcde8-67ae-4bfa-97bd-f6166cc1471d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph" Sep 10 15:33:10.106571 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.106353 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:33:10.106571 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.106448 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:10.606408702 +0000 UTC m=+49.168717903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : secret "insights-runtime-extractor-tls" not found Sep 10 15:33:10.106571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.106463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/028c54f4-e5fc-4f58-9127-390792d644ef-data-volume\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.106571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.106469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.106571 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.106563 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:33:10.106895 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.106615 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:10.606600517 +0000 UTC m=+49.168909702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : configmap "kube-rbac-proxy" not found Sep 10 15:33:10.114928 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.114893 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfz4b\" (UniqueName: \"kubernetes.io/projected/325fcde8-67ae-4bfa-97bd-f6166cc1471d-kube-api-access-bfz4b\") pod \"migrator-fdbb8fdff-wv5ph\" (UID: \"325fcde8-67ae-4bfa-97bd-f6166cc1471d\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph" Sep 10 15:33:10.115179 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.115154 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2snn\" (UniqueName: \"kubernetes.io/projected/028c54f4-e5fc-4f58-9127-390792d644ef-kube-api-access-g2snn\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.188808 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.188771 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph" Sep 10 15:33:10.611274 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.611235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.611459 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.611343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:10.611459 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.611358 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:33:10.611459 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.611436 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:11.611408519 +0000 UTC m=+50.173717687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : configmap "kube-rbac-proxy" not found Sep 10 15:33:10.611571 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.611469 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:33:10.611571 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.611517 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:11.611503958 +0000 UTC m=+50.173813151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : secret "insights-runtime-extractor-tls" not found Sep 10 15:33:10.813203 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:10.813169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:10.813358 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.813306 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 15:33:10.813401 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:10.813373 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert podName:909ac02b-6554-404c-8b05-572d1d5902a9 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:12.81335742 +0000 UTC m=+51.375666593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5m5tq" (UID: "909ac02b-6554-404c-8b05-572d1d5902a9") : secret "networking-console-plugin-cert" not found Sep 10 15:33:11.107957 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.107925 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph"] Sep 10 15:33:11.111336 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:11.111313 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325fcde8_67ae_4bfa_97bd_f6166cc1471d.slice/crio-da173d4265aaf1d608af32002fd7aaf68214d99eaa10d159503a17dbd562bd3d WatchSource:0}: Error finding container da173d4265aaf1d608af32002fd7aaf68214d99eaa10d159503a17dbd562bd3d: Status 404 returned error can't find the container with id da173d4265aaf1d608af32002fd7aaf68214d99eaa10d159503a17dbd562bd3d Sep 10 15:33:11.335072 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.335037 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz" event={"ID":"ab259f57-795d-463b-b058-ccfe8d326b3d","Type":"ContainerStarted","Data":"25d82fb42fe3a4690e71d4a8288f1f83e093d5ed5157babe8e32336d0df84060"} Sep 10 15:33:11.336651 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.336622 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-sz8sl" event={"ID":"e45a16ee-815d-4e40-b619-76a671086c16","Type":"ContainerStarted","Data":"f1785bc48a8036f8d38e58d923f407930c0382c5aecf3bdda876942cb002b1f9"} Sep 10 15:33:11.338460 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.338435 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/0.log" Sep 10 15:33:11.338603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.338480 2580 generic.go:356] "Generic (PLEG): container finished" podID="ecc661e5-071a-44c4-a08d-7cf930601269" containerID="05d06ebc3ae7920dd55c007006c00ecd924c4f5d53e4847e9938429faa5344aa" exitCode=255 Sep 10 15:33:11.338603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.338515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zkn28" event={"ID":"ecc661e5-071a-44c4-a08d-7cf930601269","Type":"ContainerDied","Data":"05d06ebc3ae7920dd55c007006c00ecd924c4f5d53e4847e9938429faa5344aa"} Sep 10 15:33:11.338758 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.338740 2580 scope.go:117] "RemoveContainer" containerID="05d06ebc3ae7920dd55c007006c00ecd924c4f5d53e4847e9938429faa5344aa" Sep 10 15:33:11.339882 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.339855 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph" event={"ID":"325fcde8-67ae-4bfa-97bd-f6166cc1471d","Type":"ContainerStarted","Data":"da173d4265aaf1d608af32002fd7aaf68214d99eaa10d159503a17dbd562bd3d"} Sep 10 15:33:11.341558 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.341534 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" event={"ID":"50a8ad50-e704-43b5-ac07-734b0a43ce0b","Type":"ContainerStarted","Data":"371c6180c73f3272bc845a95dcc2bc388c898df9fd6e245ba3ed49fabfae4d44"} Sep 10 15:33:11.351952 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.351896 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-cvzzz" podStartSLOduration=1.765118103 podStartE2EDuration="6.351878362s" totalImagesPullingTime="4.586760259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:11.350273406 +0000 UTC m=+49.912582595" watchObservedRunningTime="2026-09-10 15:33:11.351878362 +0000 UTC m=+49.914187550" Sep 10 15:33:11.365292 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.365206 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-sz8sl" podStartSLOduration=1.749494739 podStartE2EDuration="6.365188463s" totalImagesPullingTime="4.615693724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:11.36479298 +0000 UTC m=+49.927102171" watchObservedRunningTime="2026-09-10 15:33:11.365188463 +0000 UTC m=+49.927497654" Sep 10 15:33:11.403319 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.403266 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" podStartSLOduration=1.848911904 podStartE2EDuration="6.403245329s" totalImagesPullingTime="4.554333425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:11.402996375 +0000 UTC m=+49.965305566" watchObservedRunningTime="2026-09-10 15:33:11.403245329 +0000 UTC m=+49.965554514" Sep 10 15:33:11.508056 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.508034 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hvffx_f636588a-67d1-4196-91d5-783f5ca11b8a/dns-node-resolver/0.log" Sep 10 15:33:11.521037 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.520994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:33:11.521147 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:11.521085 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:33:11.521204 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:11.521153 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls podName:fa218d1e-4464-49e6-b1e8-141f0dd3aace nodeName:}" failed. No retries permitted until 2026-09-10 15:33:27.521135568 +0000 UTC m=+66.083444749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls") pod "dns-default-m5dct" (UID: "fa218d1e-4464-49e6-b1e8-141f0dd3aace") : secret "dns-default-metrics-tls" not found Sep 10 15:33:11.622120 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.622008 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:11.622120 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:11.622111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:11.622356 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:11.622157 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:33:11.622356 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:11.622228 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:33:11.622356 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:11.622242 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:13.622219506 +0000 UTC m=+52.184528692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : secret "insights-runtime-extractor-tls" not found Sep 10 15:33:11.622356 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:11.622283 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:13.622266244 +0000 UTC m=+52.184575412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : configmap "kube-rbac-proxy" not found Sep 10 15:33:12.108463 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:12.108380 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6hvnc_b92f14a2-c880-4843-98be-7a8480914160/node-ca/0.log" Sep 10 15:33:12.346930 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:12.346905 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:33:12.347358 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:12.347340 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/0.log" Sep 10 15:33:12.347447 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:12.347384 2580 generic.go:356] "Generic (PLEG): container finished" podID="ecc661e5-071a-44c4-a08d-7cf930601269" containerID="c5d772929aae03a2beb3dba8bb34ee551a7a93be31191f81a0aab44c1bc590e3" exitCode=255 Sep 10 15:33:12.347500 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:12.347470 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zkn28" event={"ID":"ecc661e5-071a-44c4-a08d-7cf930601269","Type":"ContainerDied","Data":"c5d772929aae03a2beb3dba8bb34ee551a7a93be31191f81a0aab44c1bc590e3"} Sep 10 15:33:12.347556 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:12.347544 2580 scope.go:117] "RemoveContainer" containerID="05d06ebc3ae7920dd55c007006c00ecd924c4f5d53e4847e9938429faa5344aa" Sep 10 15:33:12.347867 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:12.347729 2580 scope.go:117] "RemoveContainer" containerID="c5d772929aae03a2beb3dba8bb34ee551a7a93be31191f81a0aab44c1bc590e3" Sep 10 15:33:12.347960 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:12.347933 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zkn28_openshift-console-operator(ecc661e5-071a-44c4-a08d-7cf930601269)\"" pod="openshift-console-operator/console-operator-85b454874b-zkn28" podUID="ecc661e5-071a-44c4-a08d-7cf930601269" Sep 10 15:33:12.833913 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:12.833874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:12.834112 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:12.834006 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 15:33:12.834112 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:12.834081 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert podName:909ac02b-6554-404c-8b05-572d1d5902a9 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:16.834061623 +0000 UTC m=+55.396370790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5m5tq" (UID: "909ac02b-6554-404c-8b05-572d1d5902a9") : secret "networking-console-plugin-cert" not found Sep 10 15:33:13.110453 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:13.110414 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-htnn6_9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd/serve-healthcheck-canary/0.log" Sep 10 15:33:13.353033 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:13.353004 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:33:13.353366 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:13.353348 2580 scope.go:117] "RemoveContainer" containerID="c5d772929aae03a2beb3dba8bb34ee551a7a93be31191f81a0aab44c1bc590e3" Sep 10 15:33:13.353571 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:13.353550 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zkn28_openshift-console-operator(ecc661e5-071a-44c4-a08d-7cf930601269)\"" pod="openshift-console-operator/console-operator-85b454874b-zkn28" podUID="ecc661e5-071a-44c4-a08d-7cf930601269" Sep 10 15:33:13.354750 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:13.354721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph" event={"ID":"325fcde8-67ae-4bfa-97bd-f6166cc1471d","Type":"ContainerStarted","Data":"6c318a942078a349666a8898b57eb3f3738f698b1d67cfcebf98f3e43a70fd26"} Sep 10 15:33:13.354859 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:13.354790 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph" event={"ID":"325fcde8-67ae-4bfa-97bd-f6166cc1471d","Type":"ContainerStarted","Data":"3f516fd2d90bbee29892225958e39cfa1890f9ee7a18c91548c2bf63a2f2b546"} Sep 10 15:33:13.393474 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:13.393355 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wv5ph" podStartSLOduration=2.904314711 podStartE2EDuration="4.393338199s" totalImagesPullingTime="1.489023488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:13.392316401 +0000 UTC m=+51.954625591" watchObservedRunningTime="2026-09-10 15:33:13.393338199 +0000 UTC m=+51.955647424" Sep 10 15:33:13.538970 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:13.538933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:13.539161 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:13.539088 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:33:13.539161 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:13.539116 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6966675f-zbjnc: secret "image-registry-tls" not found Sep 10 15:33:13.539275 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:13.539181 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls podName:d1d7ffd5-c3f4-4e66-a1be-aa7be0310165 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:21.53916217 +0000 UTC m=+60.101471338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls") pod "image-registry-6d6966675f-zbjnc" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165") : secret "image-registry-tls" not found Sep 10 15:33:13.639748 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:13.639706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:13.639944 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:13.639795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:13.639944 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:13.639841 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:33:13.639944 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:13.639879 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:33:13.639944 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:13.639906 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:17.639891026 +0000 UTC m=+56.202200195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : secret "insights-runtime-extractor-tls" not found Sep 10 15:33:13.639944 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:13.639926 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:17.639912533 +0000 UTC m=+56.202221707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : configmap "kube-rbac-proxy" not found Sep 10 15:33:16.035650 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:16.035615 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:16.036097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:16.035698 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:16.036097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:16.035980 2580 scope.go:117] "RemoveContainer" containerID="c5d772929aae03a2beb3dba8bb34ee551a7a93be31191f81a0aab44c1bc590e3" Sep 10 15:33:16.036181 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:16.036119 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zkn28_openshift-console-operator(ecc661e5-071a-44c4-a08d-7cf930601269)\"" pod="openshift-console-operator/console-operator-85b454874b-zkn28" podUID="ecc661e5-071a-44c4-a08d-7cf930601269" Sep 10 15:33:16.365973 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:16.365947 2580 scope.go:117] "RemoveContainer" containerID="c5d772929aae03a2beb3dba8bb34ee551a7a93be31191f81a0aab44c1bc590e3" Sep 10 15:33:16.366454 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:16.366406 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-zkn28_openshift-console-operator(ecc661e5-071a-44c4-a08d-7cf930601269)\"" pod="openshift-console-operator/console-operator-85b454874b-zkn28" podUID="ecc661e5-071a-44c4-a08d-7cf930601269" Sep 10 15:33:16.862923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:16.862885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:16.863079 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:16.863002 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 15:33:16.863079 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:16.863057 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert podName:909ac02b-6554-404c-8b05-572d1d5902a9 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:24.863042931 +0000 UTC m=+63.425352100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5m5tq" (UID: "909ac02b-6554-404c-8b05-572d1d5902a9") : secret "networking-console-plugin-cert" not found Sep 10 15:33:17.669074 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:17.669030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:17.669475 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:17.669101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:17.669475 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:17.669154 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:33:17.669475 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:17.669180 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:33:17.669475 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:17.669204 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:25.669190804 +0000 UTC m=+64.231499973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : configmap "kube-rbac-proxy" not found Sep 10 15:33:17.669475 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:17.669238 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:25.669223272 +0000 UTC m=+64.231532440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : secret "insights-runtime-extractor-tls" not found Sep 10 15:33:21.603133 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:21.603091 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:21.603626 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:21.603247 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:33:21.603626 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:21.603269 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d6966675f-zbjnc: secret "image-registry-tls" not found Sep 10 15:33:21.603626 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:21.603326 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls podName:d1d7ffd5-c3f4-4e66-a1be-aa7be0310165 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:37.603309244 +0000 UTC m=+76.165618412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls") pod "image-registry-6d6966675f-zbjnc" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165") : secret "image-registry-tls" not found Sep 10 15:33:22.274167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:22.274129 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g68fj" Sep 10 15:33:24.929351 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:24.929307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:24.929852 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:24.929533 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 15:33:24.929852 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:24.929623 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert podName:909ac02b-6554-404c-8b05-572d1d5902a9 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:40.929600697 +0000 UTC m=+79.491909880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-5m5tq" (UID: "909ac02b-6554-404c-8b05-572d1d5902a9") : secret "networking-console-plugin-cert" not found Sep 10 15:33:25.735273 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:25.735232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:25.735483 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:25.735338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:25.735483 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:25.735464 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:33:25.735562 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:25.735515 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls podName:028c54f4-e5fc-4f58-9127-390792d644ef nodeName:}" failed. No retries permitted until 2026-09-10 15:33:41.735502981 +0000 UTC m=+80.297812148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w8np8" (UID: "028c54f4-e5fc-4f58-9127-390792d644ef") : secret "insights-runtime-extractor-tls" not found Sep 10 15:33:25.735766 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:25.735744 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028c54f4-e5fc-4f58-9127-390792d644ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:27.549530 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.549481 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:33:27.551755 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.551734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa218d1e-4464-49e6-b1e8-141f0dd3aace-metrics-tls\") pod \"dns-default-m5dct\" (UID: \"fa218d1e-4464-49e6-b1e8-141f0dd3aace\") " pod="openshift-dns/dns-default-m5dct" Sep 10 15:33:27.751368 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.751331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:33:27.751368 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.751376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:33:27.754229 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.754209 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 15:33:27.754986 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.754968 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 15:33:27.763816 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.763792 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f044f5a4-1d02-48b3-a5c3-0dc9f2212f51-original-pull-secret\") pod \"global-pull-secret-syncer-8ld8d\" (UID: \"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51\") " pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:33:27.763892 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.763842 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6150e082-626a-47a0-b8a0-5a9ea64b8251-metrics-certs\") pod \"network-metrics-daemon-hvz5r\" (UID: \"6150e082-626a-47a0-b8a0-5a9ea64b8251\") " pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:33:27.827265 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.827182 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bddbj\"" Sep 10 15:33:27.835126 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.835101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m5dct" Sep 10 15:33:27.837108 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.837082 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gcptc\"" Sep 10 15:33:27.845398 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.845374 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hvz5r" Sep 10 15:33:27.848735 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.848719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8ld8d" Sep 10 15:33:27.954073 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.954043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:33:27.956610 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:27.956579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfr9\" (UniqueName: \"kubernetes.io/projected/8d47d830-d5d0-45ff-92d5-c605bd23d46f-kube-api-access-8mfr9\") pod \"network-check-target-q8p4v\" (UID: \"8d47d830-d5d0-45ff-92d5-c605bd23d46f\") " pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:33:28.106605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:28.106524 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m5dct"] Sep 10 15:33:28.109607 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:28.109570 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa218d1e_4464_49e6_b1e8_141f0dd3aace.slice/crio-9d9825ee6ca53502e55b333f53bc57f09e7a419ba9a0d3b6b51adbb4b46d8d5b WatchSource:0}: Error finding container 9d9825ee6ca53502e55b333f53bc57f09e7a419ba9a0d3b6b51adbb4b46d8d5b: Status 404 returned error can't find the container with id 9d9825ee6ca53502e55b333f53bc57f09e7a419ba9a0d3b6b51adbb4b46d8d5b Sep 10 15:33:28.128074 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:28.128049 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8ld8d"] Sep 10 15:33:28.131530 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:28.131505 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf044f5a4_1d02_48b3_a5c3_0dc9f2212f51.slice/crio-d773c3166ab756e457e346688bc967e8f5c11573acb2e5623e2b569e6abf204f WatchSource:0}: Error finding container d773c3166ab756e457e346688bc967e8f5c11573acb2e5623e2b569e6abf204f: Status 404 returned error can't find the container with id d773c3166ab756e457e346688bc967e8f5c11573acb2e5623e2b569e6abf204f Sep 10 15:33:28.132054 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:28.132030 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hvz5r"] Sep 10 15:33:28.135415 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:28.135395 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6150e082_626a_47a0_b8a0_5a9ea64b8251.slice/crio-4dc75a07ff95becc2a33ed0e15347759a8d0d44a40149108cbea0e2e57e30931 WatchSource:0}: Error finding container 4dc75a07ff95becc2a33ed0e15347759a8d0d44a40149108cbea0e2e57e30931: Status 404 returned error can't find the container with id 4dc75a07ff95becc2a33ed0e15347759a8d0d44a40149108cbea0e2e57e30931 Sep 10 15:33:28.143672 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:28.143655 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-skj7z\"" Sep 10 15:33:28.152491 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:28.152471 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:33:28.404762 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:28.404725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hvz5r" event={"ID":"6150e082-626a-47a0-b8a0-5a9ea64b8251","Type":"ContainerStarted","Data":"4dc75a07ff95becc2a33ed0e15347759a8d0d44a40149108cbea0e2e57e30931"} Sep 10 15:33:28.405907 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:28.405884 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5dct" event={"ID":"fa218d1e-4464-49e6-b1e8-141f0dd3aace","Type":"ContainerStarted","Data":"9d9825ee6ca53502e55b333f53bc57f09e7a419ba9a0d3b6b51adbb4b46d8d5b"} Sep 10 15:33:28.406931 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:28.406905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8ld8d" event={"ID":"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51","Type":"ContainerStarted","Data":"d773c3166ab756e457e346688bc967e8f5c11573acb2e5623e2b569e6abf204f"} Sep 10 15:33:28.422898 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:28.422874 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q8p4v"] Sep 10 15:33:28.426789 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:28.426767 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d47d830_d5d0_45ff_92d5_c605bd23d46f.slice/crio-9b34bd8c2a40043e4af8da1f2987ec7732dfb4b226605d64aaeb954414b870df WatchSource:0}: Error finding container 9b34bd8c2a40043e4af8da1f2987ec7732dfb4b226605d64aaeb954414b870df: Status 404 returned error can't find the container with id 9b34bd8c2a40043e4af8da1f2987ec7732dfb4b226605d64aaeb954414b870df Sep 10 15:33:29.412720 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:29.412693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8p4v" event={"ID":"8d47d830-d5d0-45ff-92d5-c605bd23d46f","Type":"ContainerStarted","Data":"67e09842fedb067872dd035ff285e4279a77a21bc29042759f1664744f6019d4"} Sep 10 15:33:29.413078 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:29.412743 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q8p4v" event={"ID":"8d47d830-d5d0-45ff-92d5-c605bd23d46f","Type":"ContainerStarted","Data":"9b34bd8c2a40043e4af8da1f2987ec7732dfb4b226605d64aaeb954414b870df"} Sep 10 15:33:29.413078 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:29.412909 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:33:29.432008 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:29.431815 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q8p4v" podStartSLOduration=67.431784098 podStartE2EDuration="1m7.431784098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:29.429766106 +0000 UTC m=+67.992075296" watchObservedRunningTime="2026-09-10 15:33:29.431784098 +0000 UTC m=+67.994093289" Sep 10 15:33:30.419158 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:30.419116 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hvz5r" event={"ID":"6150e082-626a-47a0-b8a0-5a9ea64b8251","Type":"ContainerStarted","Data":"e2e99028edccd3175ed28f5e3b9ba42cc996c06bff5e181f7b9990a65e16d348"} Sep 10 15:33:30.419654 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:30.419184 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hvz5r" event={"ID":"6150e082-626a-47a0-b8a0-5a9ea64b8251","Type":"ContainerStarted","Data":"09695fff32b0a44792283d5c3ee05cc8d3df89bcd4e9a9d82f3a4212f88d18c2"} Sep 10 15:33:30.421251 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:30.421221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5dct" event={"ID":"fa218d1e-4464-49e6-b1e8-141f0dd3aace","Type":"ContainerStarted","Data":"5daea61eb6ce0bafac4deaa6151eff88bf2069d5108100ea94cb57ca72750e2f"} Sep 10 15:33:30.421365 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:30.421286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m5dct" event={"ID":"fa218d1e-4464-49e6-b1e8-141f0dd3aace","Type":"ContainerStarted","Data":"38e76b790317cc4e907185faa4a84900efea47f10720e8c481379609426e7c6e"} Sep 10 15:33:30.436254 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:30.436197 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hvz5r" podStartSLOduration=67.273821027 podStartE2EDuration="1m8.436182503s" totalImagesPullingTime="1.162361476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:30.434840493 +0000 UTC m=+68.997149684" watchObservedRunningTime="2026-09-10 15:33:30.436182503 +0000 UTC m=+68.998491693" Sep 10 15:33:30.454225 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:30.454176 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m5dct" podStartSLOduration=33.66658508 podStartE2EDuration="35.45416058s" totalImagesPullingTime="1.7875755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:30.452696826 +0000 UTC m=+69.015006017" watchObservedRunningTime="2026-09-10 15:33:30.45416058 +0000 UTC m=+69.016469770" Sep 10 15:33:31.022207 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:31.022175 2580 scope.go:117] "RemoveContainer" containerID="c5d772929aae03a2beb3dba8bb34ee551a7a93be31191f81a0aab44c1bc590e3" Sep 10 15:33:31.426523 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:31.426487 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m5dct" Sep 10 15:33:32.430633 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:32.430608 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:33:32.431012 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:32.430688 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-zkn28" event={"ID":"ecc661e5-071a-44c4-a08d-7cf930601269","Type":"ContainerStarted","Data":"dfc60ed69d12f003bae75583c92a1bf80b5666c7602fceb95dbe34234f29dcd3"} Sep 10 15:33:32.431155 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:32.431138 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:32.432540 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:32.432512 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8ld8d" event={"ID":"f044f5a4-1d02-48b3-a5c3-0dc9f2212f51","Type":"ContainerStarted","Data":"0e292169bf06cff505d65c17dc6a66af61fdaa85bea63e6083c75630cf129e73"} Sep 10 15:33:32.448188 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:32.448144 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-zkn28" podStartSLOduration=22.940127597 podStartE2EDuration="27.448131973s" totalImagesPullingTime="4.508004376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:32.447157012 +0000 UTC m=+71.009466203" watchObservedRunningTime="2026-09-10 15:33:32.448131973 +0000 UTC m=+71.010441161" Sep 10 15:33:32.462513 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:32.462469 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8ld8d" podStartSLOduration=66.539194466 podStartE2EDuration="1m10.462457028s" totalImagesPullingTime="3.923262562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:32.462313602 +0000 UTC m=+71.024622793" watchObservedRunningTime="2026-09-10 15:33:32.462457028 +0000 UTC m=+71.024766219" Sep 10 15:33:33.034677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:33.034368 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-zkn28" Sep 10 15:33:37.633543 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:37.633509 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:37.636031 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:37.636007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"image-registry-6d6966675f-zbjnc\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:37.823212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:37.823178 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wjsnb\"" Sep 10 15:33:37.831012 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:37.830992 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:38.120999 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.120966 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d6966675f-zbjnc"] Sep 10 15:33:38.125341 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:38.125312 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d7ffd5_c3f4_4e66_a1be_aa7be0310165.slice/crio-443cbb05c529a9413a90207a12f8f2f6a027385c8123c734d0d66a8f7ba9800b WatchSource:0}: Error finding container 443cbb05c529a9413a90207a12f8f2f6a027385c8123c734d0d66a8f7ba9800b: Status 404 returned error can't find the container with id 443cbb05c529a9413a90207a12f8f2f6a027385c8123c734d0d66a8f7ba9800b Sep 10 15:33:38.453459 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.453367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" event={"ID":"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165","Type":"ContainerStarted","Data":"6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e"} Sep 10 15:33:38.453459 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.453418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" event={"ID":"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165","Type":"ContainerStarted","Data":"443cbb05c529a9413a90207a12f8f2f6a027385c8123c734d0d66a8f7ba9800b"} Sep 10 15:33:38.453694 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.453481 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:33:38.474952 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.474885 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" podStartSLOduration=33.474867662 podStartE2EDuration="33.474867662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:38.473915518 +0000 UTC m=+77.036224733" watchObservedRunningTime="2026-09-10 15:33:38.474867662 +0000 UTC m=+77.037176868" Sep 10 15:33:38.912125 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.912097 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-bzktv"] Sep 10 15:33:38.918032 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.918013 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-bzktv" Sep 10 15:33:38.920587 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.920566 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 15:33:38.921026 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.921008 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 15:33:38.921121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.921010 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tp9zh\"" Sep 10 15:33:38.942259 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.942236 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-bzktv"] Sep 10 15:33:38.951314 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:38.951293 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d6966675f-zbjnc"] Sep 10 15:33:39.014762 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.014731 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-596fd9b478-qwskh"] Sep 10 15:33:39.018365 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.018346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.033564 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.033539 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596fd9b478-qwskh"] Sep 10 15:33:39.044208 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.044177 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj67t\" (UniqueName: \"kubernetes.io/projected/a73adce0-f5ba-43c0-973d-ced40480a9a9-kube-api-access-xj67t\") pod \"downloads-85d98c998f-bzktv\" (UID: \"a73adce0-f5ba-43c0-973d-ced40480a9a9\") " pod="openshift-console/downloads-85d98c998f-bzktv" Sep 10 15:33:39.145268 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.145231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj67t\" (UniqueName: \"kubernetes.io/projected/a73adce0-f5ba-43c0-973d-ced40480a9a9-kube-api-access-xj67t\") pod \"downloads-85d98c998f-bzktv\" (UID: \"a73adce0-f5ba-43c0-973d-ced40480a9a9\") " pod="openshift-console/downloads-85d98c998f-bzktv" Sep 10 15:33:39.145268 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.145281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b04946a6-b69a-4e35-8790-c656d7747513-ca-trust-extracted\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.145499 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.145301 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b04946a6-b69a-4e35-8790-c656d7747513-trusted-ca\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.145499 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.145340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b04946a6-b69a-4e35-8790-c656d7747513-installation-pull-secrets\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.145499 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.145360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b04946a6-b69a-4e35-8790-c656d7747513-registry-tls\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.145499 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.145412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b04946a6-b69a-4e35-8790-c656d7747513-registry-certificates\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.145499 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.145455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b04946a6-b69a-4e35-8790-c656d7747513-image-registry-private-configuration\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.145499 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.145479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b04946a6-b69a-4e35-8790-c656d7747513-bound-sa-token\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.145499 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.145497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ksld\" (UniqueName: \"kubernetes.io/projected/b04946a6-b69a-4e35-8790-c656d7747513-kube-api-access-2ksld\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.159544 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.159505 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj67t\" (UniqueName: \"kubernetes.io/projected/a73adce0-f5ba-43c0-973d-ced40480a9a9-kube-api-access-xj67t\") pod \"downloads-85d98c998f-bzktv\" (UID: \"a73adce0-f5ba-43c0-973d-ced40480a9a9\") " pod="openshift-console/downloads-85d98c998f-bzktv" Sep 10 15:33:39.226533 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.226442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-bzktv" Sep 10 15:33:39.246137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.246107 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b04946a6-b69a-4e35-8790-c656d7747513-ca-trust-extracted\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.246246 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.246142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b04946a6-b69a-4e35-8790-c656d7747513-trusted-ca\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.246246 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.246182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b04946a6-b69a-4e35-8790-c656d7747513-installation-pull-secrets\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.246246 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.246203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b04946a6-b69a-4e35-8790-c656d7747513-registry-tls\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.246390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.246244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b04946a6-b69a-4e35-8790-c656d7747513-registry-certificates\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.246390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.246277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b04946a6-b69a-4e35-8790-c656d7747513-image-registry-private-configuration\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.246390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.246306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b04946a6-b69a-4e35-8790-c656d7747513-bound-sa-token\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.246390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.246335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ksld\" (UniqueName: \"kubernetes.io/projected/b04946a6-b69a-4e35-8790-c656d7747513-kube-api-access-2ksld\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.247267 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.247210 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b04946a6-b69a-4e35-8790-c656d7747513-registry-certificates\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.247372 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.247343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b04946a6-b69a-4e35-8790-c656d7747513-trusted-ca\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.247645 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.247622 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b04946a6-b69a-4e35-8790-c656d7747513-ca-trust-extracted\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.248885 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.248863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b04946a6-b69a-4e35-8790-c656d7747513-installation-pull-secrets\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.248980 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.248925 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b04946a6-b69a-4e35-8790-c656d7747513-image-registry-private-configuration\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.249161 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.249140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b04946a6-b69a-4e35-8790-c656d7747513-registry-tls\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.257744 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.257708 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b04946a6-b69a-4e35-8790-c656d7747513-bound-sa-token\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.258140 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.258120 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ksld\" (UniqueName: \"kubernetes.io/projected/b04946a6-b69a-4e35-8790-c656d7747513-kube-api-access-2ksld\") pod \"image-registry-596fd9b478-qwskh\" (UID: \"b04946a6-b69a-4e35-8790-c656d7747513\") " pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.326841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.326801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:39.502058 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.501984 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-bzktv"] Sep 10 15:33:39.504944 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:39.504906 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73adce0_f5ba_43c0_973d_ced40480a9a9.slice/crio-a25ddd2bbf05e9a75801cec6aec1578c2c23f9dcf7a926abf9d14f9ef7252894 WatchSource:0}: Error finding container a25ddd2bbf05e9a75801cec6aec1578c2c23f9dcf7a926abf9d14f9ef7252894: Status 404 returned error can't find the container with id a25ddd2bbf05e9a75801cec6aec1578c2c23f9dcf7a926abf9d14f9ef7252894 Sep 10 15:33:39.600405 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:39.600372 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-596fd9b478-qwskh"] Sep 10 15:33:39.603222 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:39.603198 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04946a6_b69a_4e35_8790_c656d7747513.slice/crio-505d4c9c6eeb2c816fd292577a9cc915555528559ce488c8ad9dc9a9a9ae82ad WatchSource:0}: Error finding container 505d4c9c6eeb2c816fd292577a9cc915555528559ce488c8ad9dc9a9a9ae82ad: Status 404 returned error can't find the container with id 505d4c9c6eeb2c816fd292577a9cc915555528559ce488c8ad9dc9a9a9ae82ad Sep 10 15:33:40.462531 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:40.462492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-bzktv" event={"ID":"a73adce0-f5ba-43c0-973d-ced40480a9a9","Type":"ContainerStarted","Data":"a25ddd2bbf05e9a75801cec6aec1578c2c23f9dcf7a926abf9d14f9ef7252894"} Sep 10 15:33:40.464417 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:40.464385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596fd9b478-qwskh" event={"ID":"b04946a6-b69a-4e35-8790-c656d7747513","Type":"ContainerStarted","Data":"51e18dab8bfcd9f15f885f5f8991f324d62d402d96e65127711b97c694803d10"} Sep 10 15:33:40.464567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:40.464478 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:33:40.464567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:40.464504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-596fd9b478-qwskh" event={"ID":"b04946a6-b69a-4e35-8790-c656d7747513","Type":"ContainerStarted","Data":"505d4c9c6eeb2c816fd292577a9cc915555528559ce488c8ad9dc9a9a9ae82ad"} Sep 10 15:33:40.485643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:40.485577 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-596fd9b478-qwskh" podStartSLOduration=2.485555566 podStartE2EDuration="2.485555566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:40.483623944 +0000 UTC m=+79.045933135" watchObservedRunningTime="2026-09-10 15:33:40.485555566 +0000 UTC m=+79.047864755" Sep 10 15:33:40.962358 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:40.962310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:40.964751 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:40.964719 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/909ac02b-6554-404c-8b05-572d1d5902a9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-5m5tq\" (UID: \"909ac02b-6554-404c-8b05-572d1d5902a9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:41.171438 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:41.171392 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9cpbh\"" Sep 10 15:33:41.179114 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:41.179090 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" Sep 10 15:33:41.435568 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:41.435537 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m5dct" Sep 10 15:33:41.479637 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:41.479608 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-5m5tq"] Sep 10 15:33:41.479977 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:41.479665 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909ac02b_6554_404c_8b05_572d1d5902a9.slice/crio-83748fc87a18cd7453c2f07d9fd8e3c129bf27eea267926263d6d895ab4a79fb WatchSource:0}: Error finding container 83748fc87a18cd7453c2f07d9fd8e3c129bf27eea267926263d6d895ab4a79fb: Status 404 returned error can't find the container with id 83748fc87a18cd7453c2f07d9fd8e3c129bf27eea267926263d6d895ab4a79fb Sep 10 15:33:41.769549 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:41.769503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:41.772289 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:41.772258 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028c54f4-e5fc-4f58-9127-390792d644ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w8np8\" (UID: \"028c54f4-e5fc-4f58-9127-390792d644ef\") " pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:41.957244 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:41.957213 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cj5pp\"" Sep 10 15:33:41.965509 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:41.965477 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w8np8" Sep 10 15:33:42.256477 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:42.256323 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w8np8"] Sep 10 15:33:42.259376 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:42.259344 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028c54f4_e5fc_4f58_9127_390792d644ef.slice/crio-ce5a17bf77f312f71b910a456e93b1e43c1f4ad6c15215268966685a16f6c2f9 WatchSource:0}: Error finding container ce5a17bf77f312f71b910a456e93b1e43c1f4ad6c15215268966685a16f6c2f9: Status 404 returned error can't find the container with id ce5a17bf77f312f71b910a456e93b1e43c1f4ad6c15215268966685a16f6c2f9 Sep 10 15:33:42.474170 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:42.474124 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" event={"ID":"909ac02b-6554-404c-8b05-572d1d5902a9","Type":"ContainerStarted","Data":"83748fc87a18cd7453c2f07d9fd8e3c129bf27eea267926263d6d895ab4a79fb"} Sep 10 15:33:42.476150 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:42.476108 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8np8" event={"ID":"028c54f4-e5fc-4f58-9127-390792d644ef","Type":"ContainerStarted","Data":"909dcbf01defb23520d54da17512bb932c4e9d34acdbd6899549155393c39ef7"} Sep 10 15:33:42.476263 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:42.476173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8np8" event={"ID":"028c54f4-e5fc-4f58-9127-390792d644ef","Type":"ContainerStarted","Data":"ce5a17bf77f312f71b910a456e93b1e43c1f4ad6c15215268966685a16f6c2f9"} Sep 10 15:33:43.482829 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.482793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" event={"ID":"909ac02b-6554-404c-8b05-572d1d5902a9","Type":"ContainerStarted","Data":"023a9fa994f3361c5a2b3c206f32caa21ec7d50007dbeb7574e91dc89ffb18cf"} Sep 10 15:33:43.485375 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.485344 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8np8" event={"ID":"028c54f4-e5fc-4f58-9127-390792d644ef","Type":"ContainerStarted","Data":"b75b2d8069a53896669aaaeff02e085e62605e0a27fa643868fae4587836ffe1"} Sep 10 15:33:43.500153 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.499944 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-5m5tq" podStartSLOduration=33.255993891 podStartE2EDuration="34.499922449s" totalImagesPullingTime="1.243928558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:43.499397257 +0000 UTC m=+82.061706448" watchObservedRunningTime="2026-09-10 15:33:43.499922449 +0000 UTC m=+82.062231639" Sep 10 15:33:43.625442 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.625288 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59b88d7448-q5p46"] Sep 10 15:33:43.628865 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.628841 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.631471 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.631446 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 15:33:43.631724 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.631708 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 15:33:43.632836 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.632739 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 15:33:43.633512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.633080 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 15:33:43.633512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.633370 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jdlj5\"" Sep 10 15:33:43.633512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.633481 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 15:33:43.640859 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.640834 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b88d7448-q5p46"] Sep 10 15:33:43.684651 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.684621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-serving-cert\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.684651 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.684657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-service-ca\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.684862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.684681 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-oauth-serving-cert\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.684862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.684783 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-oauth-config\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.684862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.684829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-config\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.684862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.684862 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtc47\" (UniqueName: \"kubernetes.io/projected/675a262c-19ad-41bb-a0e5-23f0f601b35f-kube-api-access-jtc47\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.785330 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.785247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtc47\" (UniqueName: \"kubernetes.io/projected/675a262c-19ad-41bb-a0e5-23f0f601b35f-kube-api-access-jtc47\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.785530 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.785407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-serving-cert\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.785530 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.785469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-service-ca\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.785530 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.785487 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-oauth-serving-cert\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.785530 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.785529 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-oauth-config\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.785743 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.785557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-config\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.786304 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.786270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-config\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.786447 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.786323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-service-ca\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.786598 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.786573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-oauth-serving-cert\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.788830 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.788797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-oauth-config\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.789275 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.789249 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-serving-cert\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.793899 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.793858 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtc47\" (UniqueName: \"kubernetes.io/projected/675a262c-19ad-41bb-a0e5-23f0f601b35f-kube-api-access-jtc47\") pod \"console-59b88d7448-q5p46\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:43.942922 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:43.942883 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:44.126504 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.126472 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-slm9x"] Sep 10 15:33:44.133060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.133030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.138743 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.138716 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 15:33:44.138983 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.138951 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 15:33:44.139122 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.139100 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 15:33:44.139183 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.139117 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 15:33:44.139359 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.139341 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 15:33:44.139943 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.139660 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ttrpv\"" Sep 10 15:33:44.141371 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.141350 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-slm9x"] Sep 10 15:33:44.188963 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.188927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/763bec0e-dd29-42ee-b84a-83684e5b358c-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.189153 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.189008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8plj\" (UniqueName: \"kubernetes.io/projected/763bec0e-dd29-42ee-b84a-83684e5b358c-kube-api-access-g8plj\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.189153 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.189056 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/763bec0e-dd29-42ee-b84a-83684e5b358c-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.189153 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.189081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/763bec0e-dd29-42ee-b84a-83684e5b358c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.249690 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.249656 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59b88d7448-q5p46"] Sep 10 15:33:44.290515 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.290471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8plj\" (UniqueName: \"kubernetes.io/projected/763bec0e-dd29-42ee-b84a-83684e5b358c-kube-api-access-g8plj\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.290686 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.290544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/763bec0e-dd29-42ee-b84a-83684e5b358c-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.290686 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.290580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/763bec0e-dd29-42ee-b84a-83684e5b358c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.290686 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.290622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/763bec0e-dd29-42ee-b84a-83684e5b358c-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.291377 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.291351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/763bec0e-dd29-42ee-b84a-83684e5b358c-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.293730 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.293704 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/763bec0e-dd29-42ee-b84a-83684e5b358c-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.294579 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.294530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/763bec0e-dd29-42ee-b84a-83684e5b358c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.301274 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.301230 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8plj\" (UniqueName: \"kubernetes.io/projected/763bec0e-dd29-42ee-b84a-83684e5b358c-kube-api-access-g8plj\") pod \"prometheus-operator-6f4946db56-slm9x\" (UID: \"763bec0e-dd29-42ee-b84a-83684e5b358c\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.447719 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:44.447631 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" Sep 10 15:33:44.750968 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:44.750892 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675a262c_19ad_41bb_a0e5_23f0f601b35f.slice/crio-848c77cfdb24bd4e00dd6aa0268bb5865eaba8ae598575a368a48458d962608d WatchSource:0}: Error finding container 848c77cfdb24bd4e00dd6aa0268bb5865eaba8ae598575a368a48458d962608d: Status 404 returned error can't find the container with id 848c77cfdb24bd4e00dd6aa0268bb5865eaba8ae598575a368a48458d962608d Sep 10 15:33:45.028698 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:45.028609 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-slm9x"] Sep 10 15:33:45.124080 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:45.124047 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763bec0e_dd29_42ee_b84a_83684e5b358c.slice/crio-dfe97705e067b030706090233d0e0d1bc23b6967e224bd1da61e41f6874f8e85 WatchSource:0}: Error finding container dfe97705e067b030706090233d0e0d1bc23b6967e224bd1da61e41f6874f8e85: Status 404 returned error can't find the container with id dfe97705e067b030706090233d0e0d1bc23b6967e224bd1da61e41f6874f8e85 Sep 10 15:33:45.496211 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:45.496176 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" event={"ID":"763bec0e-dd29-42ee-b84a-83684e5b358c","Type":"ContainerStarted","Data":"dfe97705e067b030706090233d0e0d1bc23b6967e224bd1da61e41f6874f8e85"} Sep 10 15:33:45.497765 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:45.497732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b88d7448-q5p46" event={"ID":"675a262c-19ad-41bb-a0e5-23f0f601b35f","Type":"ContainerStarted","Data":"848c77cfdb24bd4e00dd6aa0268bb5865eaba8ae598575a368a48458d962608d"} Sep 10 15:33:45.500273 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:45.500247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w8np8" event={"ID":"028c54f4-e5fc-4f58-9127-390792d644ef","Type":"ContainerStarted","Data":"223ee4a030c8f3caf5ea193949ffac7c3f05a73468044807018e77406f15346d"} Sep 10 15:33:45.520741 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:45.520528 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w8np8" podStartSLOduration=33.763090497 podStartE2EDuration="36.520507987s" totalImagesPullingTime="2.75741749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:33:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:45.519908589 +0000 UTC m=+84.082217780" watchObservedRunningTime="2026-09-10 15:33:45.520507987 +0000 UTC m=+84.082817180" Sep 10 15:33:47.512167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:47.512107 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" event={"ID":"763bec0e-dd29-42ee-b84a-83684e5b358c","Type":"ContainerStarted","Data":"e56b6e7335c01be8f94bfb6404ee7bb755ef43d3a96bf136c44c3ef1403c7b23"} Sep 10 15:33:47.512646 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:47.512171 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" event={"ID":"763bec0e-dd29-42ee-b84a-83684e5b358c","Type":"ContainerStarted","Data":"5ecc0b7294d1d7f8a9e969ba0a8692bc707ed3aa372c3143ed249fe0a7f8aa85"} Sep 10 15:33:47.533557 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:47.533476 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-slm9x" podStartSLOduration=1.996068772 podStartE2EDuration="3.533453726s" totalImagesPullingTime="1.537384954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:47.531361241 +0000 UTC m=+86.093670432" watchObservedRunningTime="2026-09-10 15:33:47.533453726 +0000 UTC m=+86.095762917" Sep 10 15:33:48.518747 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:48.518706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b88d7448-q5p46" event={"ID":"675a262c-19ad-41bb-a0e5-23f0f601b35f","Type":"ContainerStarted","Data":"b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9"} Sep 10 15:33:48.537010 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:48.536962 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59b88d7448-q5p46" podStartSLOduration=2.001312561 podStartE2EDuration="5.536946339s" totalImagesPullingTime="3.535633778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:48.535775451 +0000 UTC m=+87.098084640" watchObservedRunningTime="2026-09-10 15:33:48.536946339 +0000 UTC m=+87.099255529" Sep 10 15:33:49.520158 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.520122 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v8hgk"] Sep 10 15:33:49.523626 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.523603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.526304 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.526284 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 15:33:49.526827 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.526807 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 15:33:49.527020 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.527003 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nmcbx\"" Sep 10 15:33:49.527288 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.527267 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 15:33:49.639629 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.639589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-textfile\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.639629 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.639634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.639872 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.639659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-sys\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.639872 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.639758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-tls\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.639872 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.639790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftp27\" (UniqueName: \"kubernetes.io/projected/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-kube-api-access-ftp27\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.639872 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.639821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.640062 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.639925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-wtmp\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.640062 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.639973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-metrics-client-ca\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.640149 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.640086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-root\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741502 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-tls\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftp27\" (UniqueName: \"kubernetes.io/projected/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-kube-api-access-ftp27\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-wtmp\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-metrics-client-ca\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-root\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-textfile\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-sys\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.741926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.741840 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-sys\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.742101 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:49.741954 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 15:33:49.742101 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:33:49.742015 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-tls podName:59d30f05-1cd1-4e1e-9f2c-31d142c62bfc nodeName:}" failed. No retries permitted until 2026-09-10 15:33:50.241994149 +0000 UTC m=+88.804303332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-tls") pod "node-exporter-v8hgk" (UID: "59d30f05-1cd1-4e1e-9f2c-31d142c62bfc") : secret "node-exporter-tls" not found Sep 10 15:33:49.742494 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.742465 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-root\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.742768 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.742717 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-wtmp\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.742863 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.742846 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-textfile\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.743184 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.743159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.743533 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.743511 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-metrics-client-ca\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.745546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.745503 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:49.751162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:49.751090 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftp27\" (UniqueName: \"kubernetes.io/projected/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-kube-api-access-ftp27\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:50.246533 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:50.246488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-tls\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:50.249300 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:50.249271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59d30f05-1cd1-4e1e-9f2c-31d142c62bfc-node-exporter-tls\") pod \"node-exporter-v8hgk\" (UID: \"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc\") " pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:50.433036 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:50.433001 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v8hgk" Sep 10 15:33:53.943529 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:53.943490 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:53.943946 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:53.943559 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:53.949895 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:53.949864 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:54.549115 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.549077 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:33:54.752612 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.752570 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d"] Sep 10 15:33:54.755223 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.755198 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.757779 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.757748 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 15:33:54.757889 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.757855 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 15:33:54.758924 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.758904 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 15:33:54.759075 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.759056 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 15:33:54.759623 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.759604 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-df72v\"" Sep 10 15:33:54.759736 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.759625 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 15:33:54.765190 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.765168 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 15:33:54.773526 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.773501 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d"] Sep 10 15:33:54.889116 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.889072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-metrics-client-ca\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.889320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.889126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-federate-client-tls\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.889320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.889159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.889320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.889196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-serving-certs-ca-bundle\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.889320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.889223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-telemeter-client-tls\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.889320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.889270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.889579 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.889327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-secret-telemeter-client\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.889579 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.889366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwt5f\" (UniqueName: \"kubernetes.io/projected/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-kube-api-access-qwt5f\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.990885 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.990844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-metrics-client-ca\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.990885 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.990893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-federate-client-tls\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.991454 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.990919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.991454 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.990948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-serving-certs-ca-bundle\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.991454 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.990979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-telemeter-client-tls\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.991454 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.991003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.991454 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.991037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-secret-telemeter-client\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.991454 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.991080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwt5f\" (UniqueName: \"kubernetes.io/projected/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-kube-api-access-qwt5f\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.991828 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.991799 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-metrics-client-ca\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.991929 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.991904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-serving-certs-ca-bundle\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.992028 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.992000 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.994032 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.994006 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-federate-client-tls\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.994171 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.994147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.994299 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.994278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-secret-telemeter-client\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:54.994700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:54.994678 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-telemeter-client-tls\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:55.004915 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.004890 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwt5f\" (UniqueName: \"kubernetes.io/projected/8f0ca232-2cd0-4ea1-ab3d-426d08c833b9-kube-api-access-qwt5f\") pod \"telemeter-client-6d6d6544c7-rdh7d\" (UID: \"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9\") " pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:55.067055 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.067020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" Sep 10 15:33:55.776166 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.775559 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:33:55.780827 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.779661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.782575 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.782527 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 15:33:55.783356 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.783200 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 15:33:55.783739 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.783713 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 15:33:55.783957 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.783872 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5qhdd\"" Sep 10 15:33:55.783957 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.783937 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1lc112i8u3nev\"" Sep 10 15:33:55.784217 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.783964 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 15:33:55.784217 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784132 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 15:33:55.784217 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784199 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 15:33:55.784505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784383 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 15:33:55.784505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 15:33:55.784505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784455 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 15:33:55.784793 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784642 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 15:33:55.784793 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784676 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 15:33:55.784995 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784799 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 15:33:55.784995 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784872 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 15:33:55.784995 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.784982 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 15:33:55.786789 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.786761 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 15:33:55.801221 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.801188 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:33:55.900587 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.900587 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.900825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.900825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.900825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.900825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900704 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.900825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.900825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.900986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.901019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.901044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.901067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.901098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901407 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.901123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmwmc\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-kube-api-access-zmwmc\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901407 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.901220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901407 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.901254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:55.901407 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:55.901292 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002246 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002246 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002238 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002246 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmwmc\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-kube-api-access-zmwmc\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.002860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.002552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.004604 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.004573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.004724 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.004629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.004724 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.004662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.004724 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.004690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.004880 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.004726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.004880 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.004760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.004880 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.004783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.005193 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.005158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.007659 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.005355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.009095 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.008893 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.009095 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.009082 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.010214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.009500 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.010214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.009570 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.010214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.009607 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-web-config\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.010214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.009845 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.010214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.010182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.010214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.010184 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.011169 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.010857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.011706 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.011386 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.012129 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.012103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.012720 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.012692 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.013057 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.013031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.013134 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.013108 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.013647 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.013621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.013924 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.013896 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config-out\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.014470 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.014411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.023079 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.023051 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmwmc\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-kube-api-access-zmwmc\") pod \"prometheus-k8s-0\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.095903 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.094980 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:33:56.488307 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:56.488269 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d30f05_1cd1_4e1e_9f2c_31d142c62bfc.slice/crio-751c093296af1759508b542b596fd38373ea630147dc81b8b1aa463ab16a831c WatchSource:0}: Error finding container 751c093296af1759508b542b596fd38373ea630147dc81b8b1aa463ab16a831c: Status 404 returned error can't find the container with id 751c093296af1759508b542b596fd38373ea630147dc81b8b1aa463ab16a831c Sep 10 15:33:56.554616 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.554582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8hgk" event={"ID":"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc","Type":"ContainerStarted","Data":"751c093296af1759508b542b596fd38373ea630147dc81b8b1aa463ab16a831c"} Sep 10 15:33:56.755104 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.755081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:33:56.757743 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:33:56.757710 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bc0afd_81f1_42c0_94a4_984b3cd5eb11.slice/crio-536642bffb434cb3e1eecd15ef0bfd63cda98fff3437da32cd666773a6d67ac4 WatchSource:0}: Error finding container 536642bffb434cb3e1eecd15ef0bfd63cda98fff3437da32cd666773a6d67ac4: Status 404 returned error can't find the container with id 536642bffb434cb3e1eecd15ef0bfd63cda98fff3437da32cd666773a6d67ac4 Sep 10 15:33:56.775620 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:56.774559 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d"] Sep 10 15:33:57.566172 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:57.565276 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-bzktv" event={"ID":"a73adce0-f5ba-43c0-973d-ced40480a9a9","Type":"ContainerStarted","Data":"687f99c483750868847bd22f52a0269642f869f31c8daf89890e442ded618c10"} Sep 10 15:33:57.566172 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:57.566081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-bzktv" Sep 10 15:33:57.568871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:57.568678 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerStarted","Data":"536642bffb434cb3e1eecd15ef0bfd63cda98fff3437da32cd666773a6d67ac4"} Sep 10 15:33:57.570202 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:57.570172 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" event={"ID":"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9","Type":"ContainerStarted","Data":"684cc8bf043b48f13527917221a44ff9ca8f28da1537fc8bfe44fd7dcd4b9f8c"} Sep 10 15:33:57.579760 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:57.579737 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-bzktv" Sep 10 15:33:57.583984 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:57.583923 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-bzktv" podStartSLOduration=2.539618307 podStartE2EDuration="19.583905942s" totalImagesPullingTime="17.044287635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:57.583009233 +0000 UTC m=+96.145318425" watchObservedRunningTime="2026-09-10 15:33:57.583905942 +0000 UTC m=+96.146215133" Sep 10 15:33:58.577695 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:58.577662 2580 generic.go:356] "Generic (PLEG): container finished" podID="59d30f05-1cd1-4e1e-9f2c-31d142c62bfc" containerID="872ffa15b2bf4165cc082077e73c9b40b0d6a2d1135bd1e622a1cab3db440f6b" exitCode=0 Sep 10 15:33:58.578107 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:58.577772 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8hgk" event={"ID":"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc","Type":"ContainerDied","Data":"872ffa15b2bf4165cc082077e73c9b40b0d6a2d1135bd1e622a1cab3db440f6b"} Sep 10 15:33:58.584179 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:58.582931 2580 generic.go:356] "Generic (PLEG): container finished" podID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerID="370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea" exitCode=0 Sep 10 15:33:58.584179 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:58.583516 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerDied","Data":"370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea"} Sep 10 15:33:59.590941 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:59.590902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8hgk" event={"ID":"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc","Type":"ContainerStarted","Data":"50164ffc26b2050d47ddf58ac42f9240bfa693fff99de7d8058920af1492b701"} Sep 10 15:33:59.591381 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:59.590961 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8hgk" event={"ID":"59d30f05-1cd1-4e1e-9f2c-31d142c62bfc","Type":"ContainerStarted","Data":"8fa1f875aca3a0a4a401b1334b45d27d40a006d722da96e7cc0e2b9063f699a3"} Sep 10 15:33:59.618498 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:33:59.617469 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v8hgk" podStartSLOduration=9.454744853 podStartE2EDuration="10.617446057s" totalImagesPullingTime="1.070900415s" totalInitContainerRuntime="91.800789ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:59.615098947 +0000 UTC m=+98.177408138" watchObservedRunningTime="2026-09-10 15:33:59.617446057 +0000 UTC m=+98.179755247" Sep 10 15:34:00.424875 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:00.424683 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q8p4v" Sep 10 15:34:00.472469 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:00.472226 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:34:00.599903 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:00.599853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" event={"ID":"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9","Type":"ContainerStarted","Data":"4cfe132c4228c911eef409afc373b62e2708e0ddc4cb9add1c5677dbaacda794"} Sep 10 15:34:00.600622 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:00.599927 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" event={"ID":"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9","Type":"ContainerStarted","Data":"0facebfcb1f119cd58a6961a70f327c5d49bf4cb9e238509cae232a93844937f"} Sep 10 15:34:00.600622 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:00.599958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" event={"ID":"8f0ca232-2cd0-4ea1-ab3d-426d08c833b9","Type":"ContainerStarted","Data":"e7e627b4b1117c52a4ac07deeacda4a894e518302b7f8245a029bfe993a432b5"} Sep 10 15:34:00.629618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:00.627944 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6d6d6544c7-rdh7d" podStartSLOduration=3.577379577 podStartE2EDuration="6.627918016s" totalImagesPullingTime="3.050538439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:00.626568632 +0000 UTC m=+99.188877823" watchObservedRunningTime="2026-09-10 15:34:00.627918016 +0000 UTC m=+99.190227208" Sep 10 15:34:01.475455 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:01.475056 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-596fd9b478-qwskh" Sep 10 15:34:03.619491 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:03.619398 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerStarted","Data":"0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a"} Sep 10 15:34:04.630981 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:04.630944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerStarted","Data":"d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13"} Sep 10 15:34:05.488546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:05.488470 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" podUID="d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" containerName="registry" containerID="cri-o://6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e" gracePeriod=30 Sep 10 15:34:06.152556 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.152523 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:34:06.208098 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208023 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ss5d\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-kube-api-access-6ss5d\") pod \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " Sep 10 15:34:06.208098 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208064 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-trusted-ca\") pod \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " Sep 10 15:34:06.208331 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208103 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-installation-pull-secrets\") pod \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " Sep 10 15:34:06.208331 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208138 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-certificates\") pod \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " Sep 10 15:34:06.208331 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208159 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-bound-sa-token\") pod \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " Sep 10 15:34:06.208331 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208234 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") pod \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " Sep 10 15:34:06.208558 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208354 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-image-registry-private-configuration\") pod \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " Sep 10 15:34:06.208558 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208392 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-ca-trust-extracted\") pod \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\" (UID: \"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165\") " Sep 10 15:34:06.208668 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208605 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:34:06.208668 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208624 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:34:06.208770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208712 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-certificates\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:06.208770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.208730 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-trusted-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:06.210784 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.210753 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:34:06.211091 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.210855 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-kube-api-access-6ss5d" (OuterVolumeSpecName: "kube-api-access-6ss5d") pod "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165"). InnerVolumeSpecName "kube-api-access-6ss5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:34:06.211091 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.211076 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:34:06.211196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.211094 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:34:06.211196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.211096 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:34:06.216782 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.216753 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" (UID: "d1d7ffd5-c3f4-4e66-a1be-aa7be0310165"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:34:06.309090 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.309048 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-image-registry-private-configuration\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:06.309090 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.309083 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-ca-trust-extracted\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:06.309090 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.309092 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ss5d\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-kube-api-access-6ss5d\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:06.309090 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.309102 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-installation-pull-secrets\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:06.309334 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.309111 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-bound-sa-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:06.309334 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.309121 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165-registry-tls\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:06.642840 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.642809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerStarted","Data":"61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a"} Sep 10 15:34:06.643007 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.642919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerStarted","Data":"c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1"} Sep 10 15:34:06.643007 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.642951 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerStarted","Data":"129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f"} Sep 10 15:34:06.644034 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.644004 2580 generic.go:356] "Generic (PLEG): container finished" podID="d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" containerID="6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e" exitCode=0 Sep 10 15:34:06.644162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.644051 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" event={"ID":"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165","Type":"ContainerDied","Data":"6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e"} Sep 10 15:34:06.644162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.644064 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" Sep 10 15:34:06.644162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.644082 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d6966675f-zbjnc" event={"ID":"d1d7ffd5-c3f4-4e66-a1be-aa7be0310165","Type":"ContainerDied","Data":"443cbb05c529a9413a90207a12f8f2f6a027385c8123c734d0d66a8f7ba9800b"} Sep 10 15:34:06.644162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.644107 2580 scope.go:117] "RemoveContainer" containerID="6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e" Sep 10 15:34:06.653462 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.653447 2580 scope.go:117] "RemoveContainer" containerID="6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e" Sep 10 15:34:06.653756 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:34:06.653733 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e\": container with ID starting with 6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e not found: ID does not exist" containerID="6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e" Sep 10 15:34:06.653918 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.653764 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e"} err="failed to get container status \"6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e\": rpc error: code = NotFound desc = could not find container \"6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e\": container with ID starting with 6c59f1b01b9b18bc32f2510a65c797ae22322f7e1e9c69f673c883ea92a81c6e not found: ID does not exist" Sep 10 15:34:06.665698 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.665671 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d6966675f-zbjnc"] Sep 10 15:34:06.669661 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:06.669639 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d6966675f-zbjnc"] Sep 10 15:34:07.651825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:07.651778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerStarted","Data":"1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1"} Sep 10 15:34:07.687911 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:07.687800 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.593160601 podStartE2EDuration="12.687779496s" totalImagesPullingTime="9.077911057s" totalInitContainerRuntime="16.707838ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:07.6865654 +0000 UTC m=+106.248874601" watchObservedRunningTime="2026-09-10 15:34:07.687779496 +0000 UTC m=+106.250088686" Sep 10 15:34:08.026963 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:08.026880 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" path="/var/lib/kubelet/pods/d1d7ffd5-c3f4-4e66-a1be-aa7be0310165/volumes" Sep 10 15:34:11.095872 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:11.095833 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:34:12.453004 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:12.452968 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b88d7448-q5p46"] Sep 10 15:34:37.474782 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.474740 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59b88d7448-q5p46" podUID="675a262c-19ad-41bb-a0e5-23f0f601b35f" containerName="console" containerID="cri-o://b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9" gracePeriod=15 Sep 10 15:34:37.737941 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.737858 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:34:37.775392 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.775356 2580 generic.go:356] "Generic (PLEG): container finished" podID="675a262c-19ad-41bb-a0e5-23f0f601b35f" containerID="b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9" exitCode=0 Sep 10 15:34:37.775605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.775403 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59b88d7448-q5p46" Sep 10 15:34:37.775605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.775406 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b88d7448-q5p46" event={"ID":"675a262c-19ad-41bb-a0e5-23f0f601b35f","Type":"ContainerDied","Data":"b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9"} Sep 10 15:34:37.775605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.775530 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59b88d7448-q5p46" event={"ID":"675a262c-19ad-41bb-a0e5-23f0f601b35f","Type":"ContainerDied","Data":"848c77cfdb24bd4e00dd6aa0268bb5865eaba8ae598575a368a48458d962608d"} Sep 10 15:34:37.775605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.775553 2580 scope.go:117] "RemoveContainer" containerID="b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9" Sep 10 15:34:37.784264 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.784243 2580 scope.go:117] "RemoveContainer" containerID="b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9" Sep 10 15:34:37.784582 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:34:37.784562 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9\": container with ID starting with b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9 not found: ID does not exist" containerID="b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9" Sep 10 15:34:37.784643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.784595 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9"} err="failed to get container status \"b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9\": rpc error: code = NotFound desc = could not find container \"b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9\": container with ID starting with b6add5af5d53ae9bf616d724ad482eeb8a74a2d987436ada4c0feb0aff02fcc9 not found: ID does not exist" Sep 10 15:34:37.879099 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.879065 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-oauth-serving-cert\") pod \"675a262c-19ad-41bb-a0e5-23f0f601b35f\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " Sep 10 15:34:37.879099 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.879101 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-oauth-config\") pod \"675a262c-19ad-41bb-a0e5-23f0f601b35f\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " Sep 10 15:34:37.879335 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.879121 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-config\") pod \"675a262c-19ad-41bb-a0e5-23f0f601b35f\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " Sep 10 15:34:37.879335 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.879150 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-serving-cert\") pod \"675a262c-19ad-41bb-a0e5-23f0f601b35f\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " Sep 10 15:34:37.879335 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.879172 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-service-ca\") pod \"675a262c-19ad-41bb-a0e5-23f0f601b35f\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " Sep 10 15:34:37.879335 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.879205 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtc47\" (UniqueName: \"kubernetes.io/projected/675a262c-19ad-41bb-a0e5-23f0f601b35f-kube-api-access-jtc47\") pod \"675a262c-19ad-41bb-a0e5-23f0f601b35f\" (UID: \"675a262c-19ad-41bb-a0e5-23f0f601b35f\") " Sep 10 15:34:37.879602 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.879574 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-config" (OuterVolumeSpecName: "console-config") pod "675a262c-19ad-41bb-a0e5-23f0f601b35f" (UID: "675a262c-19ad-41bb-a0e5-23f0f601b35f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:34:37.879653 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.879631 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "675a262c-19ad-41bb-a0e5-23f0f601b35f" (UID: "675a262c-19ad-41bb-a0e5-23f0f601b35f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:34:37.879653 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.879641 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-service-ca" (OuterVolumeSpecName: "service-ca") pod "675a262c-19ad-41bb-a0e5-23f0f601b35f" (UID: "675a262c-19ad-41bb-a0e5-23f0f601b35f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:34:37.881418 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.881393 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/675a262c-19ad-41bb-a0e5-23f0f601b35f-kube-api-access-jtc47" (OuterVolumeSpecName: "kube-api-access-jtc47") pod "675a262c-19ad-41bb-a0e5-23f0f601b35f" (UID: "675a262c-19ad-41bb-a0e5-23f0f601b35f"). InnerVolumeSpecName "kube-api-access-jtc47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:34:37.881512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.881457 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "675a262c-19ad-41bb-a0e5-23f0f601b35f" (UID: "675a262c-19ad-41bb-a0e5-23f0f601b35f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:34:37.881569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.881544 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "675a262c-19ad-41bb-a0e5-23f0f601b35f" (UID: "675a262c-19ad-41bb-a0e5-23f0f601b35f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:34:37.980712 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.980669 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-oauth-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:37.980712 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.980705 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:37.980712 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.980714 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/675a262c-19ad-41bb-a0e5-23f0f601b35f-console-serving-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:37.980712 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.980722 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:37.980983 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.980731 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtc47\" (UniqueName: \"kubernetes.io/projected/675a262c-19ad-41bb-a0e5-23f0f601b35f-kube-api-access-jtc47\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:37.980983 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:37.980739 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/675a262c-19ad-41bb-a0e5-23f0f601b35f-oauth-serving-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:34:38.090970 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:38.090937 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59b88d7448-q5p46"] Sep 10 15:34:38.095480 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:38.095454 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59b88d7448-q5p46"] Sep 10 15:34:40.027318 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:40.027278 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="675a262c-19ad-41bb-a0e5-23f0f601b35f" path="/var/lib/kubelet/pods/675a262c-19ad-41bb-a0e5-23f0f601b35f/volumes" Sep 10 15:34:40.357938 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:40.357856 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-htnn6_9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd/serve-healthcheck-canary/0.log" Sep 10 15:34:47.815017 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:47.814972 2580 generic.go:356] "Generic (PLEG): container finished" podID="50a8ad50-e704-43b5-ac07-734b0a43ce0b" containerID="371c6180c73f3272bc845a95dcc2bc388c898df9fd6e245ba3ed49fabfae4d44" exitCode=0 Sep 10 15:34:47.815481 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:47.815046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" event={"ID":"50a8ad50-e704-43b5-ac07-734b0a43ce0b","Type":"ContainerDied","Data":"371c6180c73f3272bc845a95dcc2bc388c898df9fd6e245ba3ed49fabfae4d44"} Sep 10 15:34:47.815481 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:47.815416 2580 scope.go:117] "RemoveContainer" containerID="371c6180c73f3272bc845a95dcc2bc388c898df9fd6e245ba3ed49fabfae4d44" Sep 10 15:34:48.821036 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:48.820999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-drb9n" event={"ID":"50a8ad50-e704-43b5-ac07-734b0a43ce0b","Type":"ContainerStarted","Data":"ed5ce89d259276e8178e120a7d75de212af21f9c9b468b6bb04afb288009dd1e"} Sep 10 15:34:56.096200 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:56.096160 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:34:56.116205 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:56.116177 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:34:56.875280 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:34:56.875249 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:14.268661 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.268578 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:35:14.270175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.269581 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1" gracePeriod=600 Sep 10 15:35:14.270175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.269608 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy-web" containerID="cri-o://c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1" gracePeriod=600 Sep 10 15:35:14.270175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.269756 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy" containerID="cri-o://61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a" gracePeriod=600 Sep 10 15:35:14.270175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.269856 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="thanos-sidecar" containerID="cri-o://129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f" gracePeriod=600 Sep 10 15:35:14.270175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.269962 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="config-reloader" containerID="cri-o://d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13" gracePeriod=600 Sep 10 15:35:14.270898 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.270596 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="prometheus" containerID="cri-o://0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a" gracePeriod=600 Sep 10 15:35:14.931459 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931409 2580 generic.go:356] "Generic (PLEG): container finished" podID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerID="1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1" exitCode=0 Sep 10 15:35:14.931459 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931451 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerDied","Data":"1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1"} Sep 10 15:35:14.931664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931524 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerDied","Data":"61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a"} Sep 10 15:35:14.931664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931453 2580 generic.go:356] "Generic (PLEG): container finished" podID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerID="61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a" exitCode=0 Sep 10 15:35:14.931664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931545 2580 generic.go:356] "Generic (PLEG): container finished" podID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerID="129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f" exitCode=0 Sep 10 15:35:14.931664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931551 2580 generic.go:356] "Generic (PLEG): container finished" podID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerID="d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13" exitCode=0 Sep 10 15:35:14.931664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerDied","Data":"129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f"} Sep 10 15:35:14.931664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931588 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerDied","Data":"d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13"} Sep 10 15:35:14.931664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931615 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerDied","Data":"0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a"} Sep 10 15:35:14.931664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:14.931558 2580 generic.go:356] "Generic (PLEG): container finished" podID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerID="0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a" exitCode=0 Sep 10 15:35:15.528966 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.528940 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:15.608356 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608278 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-2\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608356 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608316 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-kube-rbac-proxy\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608356 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608337 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-grpc-tls\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608356 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608356 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmwmc\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-kube-api-access-zmwmc\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608460 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-web-config\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608493 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608525 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-serving-certs-ca-bundle\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608566 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-0\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608598 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-thanos-prometheus-http-client-file\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608630 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-metrics-client-ca\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608653 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-tls\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.608700 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608680 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-kubelet-serving-ca-bundle\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.609087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608722 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:35:15.609087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608735 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config-out\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.609087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608766 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-tls-assets\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.609087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608816 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-metrics-client-certs\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.609087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608841 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.609087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608872 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-trusted-ca-bundle\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.609087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608901 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.609087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608941 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-db\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.609087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.608973 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-1\") pod \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\" (UID: \"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11\") " Sep 10 15:35:15.609554 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.609121 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:35:15.609554 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.609259 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.609554 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.609280 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.610041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.610015 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:35:15.610666 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.610312 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:35:15.611097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.610989 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:35:15.611503 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.611291 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:35:15.612757 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.612728 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-kube-api-access-zmwmc" (OuterVolumeSpecName: "kube-api-access-zmwmc") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "kube-api-access-zmwmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:35:15.612978 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.612949 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:35:15.613562 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.613341 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:35:15.613562 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.613461 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:35:15.613562 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.613509 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:35:15.613562 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.613534 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:35:15.613821 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.613582 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:35:15.614011 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.613969 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:35:15.614514 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.614474 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:35:15.614893 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.614859 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:35:15.615056 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.615033 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config-out" (OuterVolumeSpecName: "config-out") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:35:15.615135 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.615107 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:35:15.615738 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.615716 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config" (OuterVolumeSpecName: "config") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:35:15.624389 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.624368 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-web-config" (OuterVolumeSpecName: "web-config") pod "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" (UID: "b1bc0afd-81f1-42c0-94a4-984b3cd5eb11"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:35:15.710637 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710593 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-web-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710637 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710628 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710637 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710642 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710654 2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710674 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-metrics-client-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710690 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710701 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710715 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config-out\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710727 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-tls-assets\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710737 2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-metrics-client-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710747 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710759 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710770 2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-config\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710782 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-db\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710793 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710805 2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-kube-rbac-proxy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710818 2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-secret-grpc-tls\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.710905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.710829 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmwmc\" (UniqueName: \"kubernetes.io/projected/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11-kube-api-access-zmwmc\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:35:15.943290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.943251 2580 generic.go:356] "Generic (PLEG): container finished" podID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerID="c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1" exitCode=0 Sep 10 15:35:15.943482 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.943340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerDied","Data":"c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1"} Sep 10 15:35:15.943482 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.943352 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:15.943482 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.943471 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b1bc0afd-81f1-42c0-94a4-984b3cd5eb11","Type":"ContainerDied","Data":"536642bffb434cb3e1eecd15ef0bfd63cda98fff3437da32cd666773a6d67ac4"} Sep 10 15:35:15.943609 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.943506 2580 scope.go:117] "RemoveContainer" containerID="1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1" Sep 10 15:35:15.951862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.951844 2580 scope.go:117] "RemoveContainer" containerID="61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a" Sep 10 15:35:15.960730 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.960702 2580 scope.go:117] "RemoveContainer" containerID="c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1" Sep 10 15:35:15.968164 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.968130 2580 scope.go:117] "RemoveContainer" containerID="129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f" Sep 10 15:35:15.968783 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.968762 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:35:15.973241 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.973222 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:35:15.975862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.975837 2580 scope.go:117] "RemoveContainer" containerID="d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13" Sep 10 15:35:15.982400 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.982386 2580 scope.go:117] "RemoveContainer" containerID="0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a" Sep 10 15:35:15.989462 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.989445 2580 scope.go:117] "RemoveContainer" containerID="370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea" Sep 10 15:35:15.996308 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.996291 2580 scope.go:117] "RemoveContainer" containerID="1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1" Sep 10 15:35:15.996605 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:35:15.996585 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1\": container with ID starting with 1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1 not found: ID does not exist" containerID="1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1" Sep 10 15:35:15.996655 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.996620 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1"} err="failed to get container status \"1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1\": rpc error: code = NotFound desc = could not find container \"1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1\": container with ID starting with 1cabf7bf8d91e14ddb496b5b317d18040b52a0d671033ac86ac76b3d4f498be1 not found: ID does not exist" Sep 10 15:35:15.996655 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.996641 2580 scope.go:117] "RemoveContainer" containerID="61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a" Sep 10 15:35:15.996891 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:35:15.996874 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a\": container with ID starting with 61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a not found: ID does not exist" containerID="61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a" Sep 10 15:35:15.996929 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.996896 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a"} err="failed to get container status \"61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a\": rpc error: code = NotFound desc = could not find container \"61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a\": container with ID starting with 61c9c4e5afcec703831166b34532cb4e35a9755cb171b51e057a03fcda78d08a not found: ID does not exist" Sep 10 15:35:15.996929 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.996912 2580 scope.go:117] "RemoveContainer" containerID="c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1" Sep 10 15:35:15.997129 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:35:15.997114 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1\": container with ID starting with c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1 not found: ID does not exist" containerID="c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1" Sep 10 15:35:15.997163 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.997131 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1"} err="failed to get container status \"c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1\": rpc error: code = NotFound desc = could not find container \"c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1\": container with ID starting with c2c13580439c02bef4d0cc5144e701de26002424c91e47f222b68d0d8062eef1 not found: ID does not exist" Sep 10 15:35:15.997163 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.997143 2580 scope.go:117] "RemoveContainer" containerID="129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f" Sep 10 15:35:15.997382 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:35:15.997365 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f\": container with ID starting with 129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f not found: ID does not exist" containerID="129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f" Sep 10 15:35:15.997454 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.997383 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f"} err="failed to get container status \"129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f\": rpc error: code = NotFound desc = could not find container \"129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f\": container with ID starting with 129672c329a8b1f4f9a0f36d15238d46f382173f3cd9601044af136f80196d5f not found: ID does not exist" Sep 10 15:35:15.997454 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.997397 2580 scope.go:117] "RemoveContainer" containerID="d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13" Sep 10 15:35:15.997664 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:35:15.997644 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13\": container with ID starting with d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13 not found: ID does not exist" containerID="d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13" Sep 10 15:35:15.997706 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.997671 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13"} err="failed to get container status \"d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13\": rpc error: code = NotFound desc = could not find container \"d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13\": container with ID starting with d140564d911a7a48fa0c48ca832bdc02fc47e2ed7842c691c52735eb93b89c13 not found: ID does not exist" Sep 10 15:35:15.997706 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.997689 2580 scope.go:117] "RemoveContainer" containerID="0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a" Sep 10 15:35:15.997916 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:35:15.997899 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a\": container with ID starting with 0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a not found: ID does not exist" containerID="0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a" Sep 10 15:35:15.997970 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.997918 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a"} err="failed to get container status \"0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a\": rpc error: code = NotFound desc = could not find container \"0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a\": container with ID starting with 0f0caf534ead30bb927b68f01d45c22bebaa895c861894a917fcbfbf800da84a not found: ID does not exist" Sep 10 15:35:15.997970 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.997933 2580 scope.go:117] "RemoveContainer" containerID="370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea" Sep 10 15:35:15.998183 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:35:15.998166 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea\": container with ID starting with 370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea not found: ID does not exist" containerID="370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea" Sep 10 15:35:15.998220 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:15.998188 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea"} err="failed to get container status \"370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea\": rpc error: code = NotFound desc = could not find container \"370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea\": container with ID starting with 370c571a89e5cb0c8cfa36430bde75f76ee507a6cf8a1cde4112eaf43b183fea not found: ID does not exist" Sep 10 15:35:16.000825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.000804 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:35:16.001088 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001076 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="config-reloader" Sep 10 15:35:16.001131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001091 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="config-reloader" Sep 10 15:35:16.001131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001101 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy-web" Sep 10 15:35:16.001131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001106 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy-web" Sep 10 15:35:16.001131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001113 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy-thanos" Sep 10 15:35:16.001131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001119 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy-thanos" Sep 10 15:35:16.001131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001125 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675a262c-19ad-41bb-a0e5-23f0f601b35f" containerName="console" Sep 10 15:35:16.001131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001131 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="675a262c-19ad-41bb-a0e5-23f0f601b35f" containerName="console" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001145 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="init-config-reloader" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001150 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="init-config-reloader" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001156 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="prometheus" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001161 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="prometheus" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001168 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001173 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001180 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="thanos-sidecar" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001185 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="thanos-sidecar" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001193 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" containerName="registry" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001198 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" containerName="registry" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001240 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="config-reloader" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001246 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="thanos-sidecar" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001253 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="675a262c-19ad-41bb-a0e5-23f0f601b35f" containerName="console" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001259 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy-thanos" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001265 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="prometheus" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001271 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001277 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" containerName="kube-rbac-proxy-web" Sep 10 15:35:16.001320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.001282 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d7ffd5-c3f4-4e66-a1be-aa7be0310165" containerName="registry" Sep 10 15:35:16.007034 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.007012 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.009705 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.009682 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 15:35:16.009791 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.009740 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5qhdd\"" Sep 10 15:35:16.010073 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.009994 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 15:35:16.010073 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010006 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 15:35:16.010073 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010024 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 15:35:16.010073 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010032 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1lc112i8u3nev\"" Sep 10 15:35:16.010349 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010077 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 15:35:16.010349 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010112 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 15:35:16.010349 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010248 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 15:35:16.010499 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010485 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 15:35:16.010545 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010519 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 15:35:16.010588 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010572 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 15:35:16.010635 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010622 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 15:35:16.010811 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.010796 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 15:35:16.011884 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.011865 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 15:35:16.014846 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.014828 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 15:35:16.018510 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.018494 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 15:35:16.021765 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.021744 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:35:16.027534 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.027486 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1bc0afd-81f1-42c0-94a4-984b3cd5eb11" path="/var/lib/kubelet/pods/b1bc0afd-81f1-42c0-94a4-984b3cd5eb11/volumes" Sep 10 15:35:16.113850 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.113815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.113850 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.113853 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-config\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114063 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.113885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114063 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.113910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-web-config\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114063 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.113927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114063 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.113943 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114063 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b126c126-5a9c-4087-8432-25962283b92b-config-out\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5qg\" (UniqueName: \"kubernetes.io/projected/b126c126-5a9c-4087-8432-25962283b92b-kube-api-access-5z5qg\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114357 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114357 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114357 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114304 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114357 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114367 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b126c126-5a9c-4087-8432-25962283b92b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.114512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.114476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.214897 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.214815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.214897 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.214859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.214897 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.214882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b126c126-5a9c-4087-8432-25962283b92b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215193 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215248 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215301 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-config\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215301 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215395 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-web-config\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215395 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215395 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215575 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215575 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b126c126-5a9c-4087-8432-25962283b92b-config-out\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215575 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215575 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215575 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5qg\" (UniqueName: \"kubernetes.io/projected/b126c126-5a9c-4087-8432-25962283b92b-kube-api-access-5z5qg\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215575 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215865 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215865 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.215865 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.218045 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.218011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b126c126-5a9c-4087-8432-25962283b92b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.218188 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.218166 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.218335 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.218311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.218440 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.218318 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.218501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.218488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.218826 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.218805 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-config\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.218947 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.218921 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.219016 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.215561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.219079 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.219017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.219638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.219606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.219819 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.219793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-web-config\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.219914 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.219893 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.220001 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.219979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.220067 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.220050 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.220156 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.220140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b126c126-5a9c-4087-8432-25962283b92b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.220858 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.220837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b126c126-5a9c-4087-8432-25962283b92b-config-out\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.221048 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.221025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.221446 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.221410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b126c126-5a9c-4087-8432-25962283b92b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.228149 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.228133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5qg\" (UniqueName: \"kubernetes.io/projected/b126c126-5a9c-4087-8432-25962283b92b-kube-api-access-5z5qg\") pod \"prometheus-k8s-0\" (UID: \"b126c126-5a9c-4087-8432-25962283b92b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.316693 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.316656 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:35:16.600458 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.600364 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 15:35:16.604143 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:35:16.604114 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb126c126_5a9c_4087_8432_25962283b92b.slice/crio-f317b315520907b4e29b38034233bd5e008e286bb858da8a39488ad62b20f102 WatchSource:0}: Error finding container f317b315520907b4e29b38034233bd5e008e286bb858da8a39488ad62b20f102: Status 404 returned error can't find the container with id f317b315520907b4e29b38034233bd5e008e286bb858da8a39488ad62b20f102 Sep 10 15:35:16.949834 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.949798 2580 generic.go:356] "Generic (PLEG): container finished" podID="b126c126-5a9c-4087-8432-25962283b92b" containerID="c95801f06b77352806feb0b1c379965c84d79ed4a46cc4f5901c802d0572e416" exitCode=0 Sep 10 15:35:16.949999 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.949845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b126c126-5a9c-4087-8432-25962283b92b","Type":"ContainerDied","Data":"c95801f06b77352806feb0b1c379965c84d79ed4a46cc4f5901c802d0572e416"} Sep 10 15:35:16.949999 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:16.949899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b126c126-5a9c-4087-8432-25962283b92b","Type":"ContainerStarted","Data":"f317b315520907b4e29b38034233bd5e008e286bb858da8a39488ad62b20f102"} Sep 10 15:35:17.956576 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:17.956540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b126c126-5a9c-4087-8432-25962283b92b","Type":"ContainerStarted","Data":"d6eda9ecfc5559db1daa53f894a2cf61b5c1d5ac480828ac2b77009d0043d793"} Sep 10 15:35:17.956921 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:17.956638 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b126c126-5a9c-4087-8432-25962283b92b","Type":"ContainerStarted","Data":"1307022a1336a00a35125b410f152fa2d32977fd1dc75f8fd74c08244a0f2592"} Sep 10 15:35:17.956921 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:17.956666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b126c126-5a9c-4087-8432-25962283b92b","Type":"ContainerStarted","Data":"3c3937559ff2167a574de828a206b3bc056ad8968990c8e89558b9c75386ecac"} Sep 10 15:35:17.956921 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:17.956689 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b126c126-5a9c-4087-8432-25962283b92b","Type":"ContainerStarted","Data":"fc2dd468239775ce4fe5485fe4720f1a2a2f2148528eac7af10fa982105572c1"} Sep 10 15:35:17.956921 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:17.956713 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b126c126-5a9c-4087-8432-25962283b92b","Type":"ContainerStarted","Data":"d6bc64dec2dd0d05fa5d0343fcbca4ec7fc052f21d1970d8817296b041ba6ef3"} Sep 10 15:35:17.956921 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:17.956736 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b126c126-5a9c-4087-8432-25962283b92b","Type":"ContainerStarted","Data":"db8982e6810dbd4b9943e534abbb7762a33280e85aa8a33034dfc94c39a554c0"} Sep 10 15:35:17.987597 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:17.987543 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.969803492 podStartE2EDuration="2.987524993s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.721501ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:35:17.985761368 +0000 UTC m=+176.548070567" watchObservedRunningTime="2026-09-10 15:35:17.987524993 +0000 UTC m=+176.549834185" Sep 10 15:35:21.317262 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:35:21.317224 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:36:16.316963 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:36:16.316923 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:36:16.332342 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:36:16.332310 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:36:17.196709 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:36:17.196668 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 15:37:17.281446 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.280598 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh"] Sep 10 15:37:17.284806 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.284782 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.291167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.290529 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 15:37:17.291167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.290702 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 15:37:17.291167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.290795 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 15:37:17.291167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.290924 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 15:37:17.291167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.290982 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m59ll\"" Sep 10 15:37:17.301439 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.301397 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh"] Sep 10 15:37:17.383938 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.383899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0180c692-a6f5-460f-a229-a6679ec710a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.383938 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.383953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0180c692-a6f5-460f-a229-a6679ec710a4-tmp\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.384163 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.383973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0180c692-a6f5-460f-a229-a6679ec710a4-manifests\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.384163 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.384038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvj4q\" (UniqueName: \"kubernetes.io/projected/0180c692-a6f5-460f-a229-a6679ec710a4-kube-api-access-jvj4q\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.384163 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.384098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0180c692-a6f5-460f-a229-a6679ec710a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.484478 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.484446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0180c692-a6f5-460f-a229-a6679ec710a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.484648 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.484494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0180c692-a6f5-460f-a229-a6679ec710a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.484648 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.484523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0180c692-a6f5-460f-a229-a6679ec710a4-tmp\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.484648 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.484545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0180c692-a6f5-460f-a229-a6679ec710a4-manifests\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.484648 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.484584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvj4q\" (UniqueName: \"kubernetes.io/projected/0180c692-a6f5-460f-a229-a6679ec710a4-kube-api-access-jvj4q\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.485057 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.485028 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0180c692-a6f5-460f-a229-a6679ec710a4-manifests\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.486716 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.486692 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0180c692-a6f5-460f-a229-a6679ec710a4-tmp\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.486957 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.486938 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0180c692-a6f5-460f-a229-a6679ec710a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.487027 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.487001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0180c692-a6f5-460f-a229-a6679ec710a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.492941 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.492910 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvj4q\" (UniqueName: \"kubernetes.io/projected/0180c692-a6f5-460f-a229-a6679ec710a4-kube-api-access-jvj4q\") pod \"opendatahub-operator-controller-manager-9f4cd445f-bv7vh\" (UID: \"0180c692-a6f5-460f-a229-a6679ec710a4\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.602622 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.602535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:17.882008 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:17.881984 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh"] Sep 10 15:37:17.884874 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:37:17.884848 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0180c692_a6f5_460f_a229_a6679ec710a4.slice/crio-ba5c9007346e32c3c944a258e2ddb6bccba355dc4a2d9061fbbfed192f5b1842 WatchSource:0}: Error finding container ba5c9007346e32c3c944a258e2ddb6bccba355dc4a2d9061fbbfed192f5b1842: Status 404 returned error can't find the container with id ba5c9007346e32c3c944a258e2ddb6bccba355dc4a2d9061fbbfed192f5b1842 Sep 10 15:37:18.406464 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:18.406410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" event={"ID":"0180c692-a6f5-460f-a229-a6679ec710a4","Type":"ContainerStarted","Data":"ba5c9007346e32c3c944a258e2ddb6bccba355dc4a2d9061fbbfed192f5b1842"} Sep 10 15:37:21.423614 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:21.423538 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" event={"ID":"0180c692-a6f5-460f-a229-a6679ec710a4","Type":"ContainerStarted","Data":"99d600b2d9fcdf7b0cec778754e2128b1a60ea0dc5d4e101dd78ae6b1f4fd79e"} Sep 10 15:37:21.933892 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:21.933859 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:37:21.936949 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:21.936919 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:37:21.945610 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:21.945586 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 15:37:22.431186 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:22.431147 2580 generic.go:356] "Generic (PLEG): container finished" podID="0180c692-a6f5-460f-a229-a6679ec710a4" containerID="99d600b2d9fcdf7b0cec778754e2128b1a60ea0dc5d4e101dd78ae6b1f4fd79e" exitCode=0 Sep 10 15:37:22.431607 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:22.431228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" event={"ID":"0180c692-a6f5-460f-a229-a6679ec710a4","Type":"ContainerDied","Data":"99d600b2d9fcdf7b0cec778754e2128b1a60ea0dc5d4e101dd78ae6b1f4fd79e"} Sep 10 15:37:23.439168 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:23.439134 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" event={"ID":"0180c692-a6f5-460f-a229-a6679ec710a4","Type":"ContainerStarted","Data":"fe5bdbcee8238b08c44cbde0e629babbba5212ea0bd3645ee87a2809937011f2"} Sep 10 15:37:23.439667 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:23.439227 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:23.473379 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:23.473325 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" podStartSLOduration=1.441356786 podStartE2EDuration="6.473308366s" totalImagesPullingTime="3.930520494s" totalInitContainerRuntime="1.101431086s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:23.47272368 +0000 UTC m=+302.035032871" watchObservedRunningTime="2026-09-10 15:37:23.473308366 +0000 UTC m=+302.035617553" Sep 10 15:37:34.445727 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:34.445693 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-bv7vh" Sep 10 15:37:43.890712 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:43.890673 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-556d846896-p5gdn"] Sep 10 15:37:43.893804 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:43.893785 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:43.896276 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:43.896258 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 15:37:43.897145 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:43.897121 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-p6p25\"" Sep 10 15:37:43.897258 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:43.897119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:37:43.897258 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:43.897189 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 15:37:43.897369 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:43.897343 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 15:37:43.897476 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:43.897459 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 15:37:43.903257 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:43.903237 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-556d846896-p5gdn"] Sep 10 15:37:44.016975 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.016941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b2ec53-fc10-49e7-8ba7-3260500777a3-cert\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.017175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.016994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7r2\" (UniqueName: \"kubernetes.io/projected/a9b2ec53-fc10-49e7-8ba7-3260500777a3-kube-api-access-jn7r2\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.017175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.017098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9b2ec53-fc10-49e7-8ba7-3260500777a3-metrics-cert\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.017175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.017147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9b2ec53-fc10-49e7-8ba7-3260500777a3-manager-config\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.118595 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.118551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9b2ec53-fc10-49e7-8ba7-3260500777a3-metrics-cert\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.118796 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.118606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9b2ec53-fc10-49e7-8ba7-3260500777a3-manager-config\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.118796 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.118633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b2ec53-fc10-49e7-8ba7-3260500777a3-cert\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.118796 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.118677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7r2\" (UniqueName: \"kubernetes.io/projected/a9b2ec53-fc10-49e7-8ba7-3260500777a3-kube-api-access-jn7r2\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.119349 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.119305 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a9b2ec53-fc10-49e7-8ba7-3260500777a3-manager-config\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.121062 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.121038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b2ec53-fc10-49e7-8ba7-3260500777a3-cert\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.121162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.121078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9b2ec53-fc10-49e7-8ba7-3260500777a3-metrics-cert\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.126200 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.126182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7r2\" (UniqueName: \"kubernetes.io/projected/a9b2ec53-fc10-49e7-8ba7-3260500777a3-kube-api-access-jn7r2\") pod \"lws-controller-manager-556d846896-p5gdn\" (UID: \"a9b2ec53-fc10-49e7-8ba7-3260500777a3\") " pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.204062 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.203971 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:44.470520 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.470447 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr"] Sep 10 15:37:44.474136 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.474109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.476798 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.476775 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 15:37:44.476980 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.476958 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 15:37:44.477072 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.477035 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jrxcj\"" Sep 10 15:37:44.477195 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.477181 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 15:37:44.484668 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.484645 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-556d846896-p5gdn"] Sep 10 15:37:44.488223 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:37:44.488201 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b2ec53_fc10_49e7_8ba7_3260500777a3.slice/crio-e4e52c57ebbeac7d1b20ee82a869eca48f4155007355e6713223b30e01e052b9 WatchSource:0}: Error finding container e4e52c57ebbeac7d1b20ee82a869eca48f4155007355e6713223b30e01e052b9: Status 404 returned error can't find the container with id e4e52c57ebbeac7d1b20ee82a869eca48f4155007355e6713223b30e01e052b9 Sep 10 15:37:44.488636 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.488618 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr"] Sep 10 15:37:44.491182 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.491161 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:37:44.528481 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.528450 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" event={"ID":"a9b2ec53-fc10-49e7-8ba7-3260500777a3","Type":"ContainerStarted","Data":"e4e52c57ebbeac7d1b20ee82a869eca48f4155007355e6713223b30e01e052b9"} Sep 10 15:37:44.622862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.622828 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.623010 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.622870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572ks\" (UniqueName: \"kubernetes.io/projected/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-kube-api-access-572ks\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.623010 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.622898 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.623010 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.622930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.623010 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.622963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.623162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.623011 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.623162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.623039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.623162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.623060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.623162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.623087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724252 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724252 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724252 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572ks\" (UniqueName: \"kubernetes.io/projected/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-kube-api-access-572ks\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724838 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724838 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724709 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724838 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.724978 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.724920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.725067 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.725050 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.726748 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.726728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.726919 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.726902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.733143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.733122 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.733352 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.733335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-572ks\" (UniqueName: \"kubernetes.io/projected/7c0b58fd-8f53-4a8a-ab2e-fd948778287d-kube-api-access-572ks\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799c54rwr\" (UID: \"7c0b58fd-8f53-4a8a-ab2e-fd948778287d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:44.785242 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:44.785210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:45.062887 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:45.062756 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr"] Sep 10 15:37:45.066069 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:37:45.066034 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0b58fd_8f53_4a8a_ab2e_fd948778287d.slice/crio-49c883bd3682c4be80b3da551df8678792135b895401a174910c8fc942941cb7 WatchSource:0}: Error finding container 49c883bd3682c4be80b3da551df8678792135b895401a174910c8fc942941cb7: Status 404 returned error can't find the container with id 49c883bd3682c4be80b3da551df8678792135b895401a174910c8fc942941cb7 Sep 10 15:37:45.535970 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:45.535924 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" event={"ID":"7c0b58fd-8f53-4a8a-ab2e-fd948778287d","Type":"ContainerStarted","Data":"49c883bd3682c4be80b3da551df8678792135b895401a174910c8fc942941cb7"} Sep 10 15:37:47.548431 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:47.548387 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" event={"ID":"a9b2ec53-fc10-49e7-8ba7-3260500777a3","Type":"ContainerStarted","Data":"3d50b01b27fff576c400e08e2ac189d8c6c44c2861279d7f9d4a3ed59ec393e0"} Sep 10 15:37:47.548988 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:47.548511 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:47.565349 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:47.565289 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" podStartSLOduration=2.08272012 podStartE2EDuration="4.565270768s" totalImagesPullingTime="2.482550648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:47.564681021 +0000 UTC m=+326.126990216" watchObservedRunningTime="2026-09-10 15:37:47.565270768 +0000 UTC m=+326.127579959" Sep 10 15:37:48.203930 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.203894 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sgxwt"] Sep 10 15:37:48.211196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.211171 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:37:48.213743 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.213717 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 15:37:48.213892 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.213849 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4h827\"" Sep 10 15:37:48.218890 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.218828 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sgxwt"] Sep 10 15:37:48.356320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.356279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zxv\" (UniqueName: \"kubernetes.io/projected/63fd10e0-82f9-444c-8302-2a31c8c1b98f-kube-api-access-r5zxv\") pod \"odh-model-controller-7998c67dc4-sgxwt\" (UID: \"63fd10e0-82f9-444c-8302-2a31c8c1b98f\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:37:48.356528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.356378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63fd10e0-82f9-444c-8302-2a31c8c1b98f-cert\") pod \"odh-model-controller-7998c67dc4-sgxwt\" (UID: \"63fd10e0-82f9-444c-8302-2a31c8c1b98f\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:37:48.457807 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.457718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63fd10e0-82f9-444c-8302-2a31c8c1b98f-cert\") pod \"odh-model-controller-7998c67dc4-sgxwt\" (UID: \"63fd10e0-82f9-444c-8302-2a31c8c1b98f\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:37:48.457982 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.457837 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zxv\" (UniqueName: \"kubernetes.io/projected/63fd10e0-82f9-444c-8302-2a31c8c1b98f-kube-api-access-r5zxv\") pod \"odh-model-controller-7998c67dc4-sgxwt\" (UID: \"63fd10e0-82f9-444c-8302-2a31c8c1b98f\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:37:48.460559 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.460530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/63fd10e0-82f9-444c-8302-2a31c8c1b98f-cert\") pod \"odh-model-controller-7998c67dc4-sgxwt\" (UID: \"63fd10e0-82f9-444c-8302-2a31c8c1b98f\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:37:48.469471 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.469445 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zxv\" (UniqueName: \"kubernetes.io/projected/63fd10e0-82f9-444c-8302-2a31c8c1b98f-kube-api-access-r5zxv\") pod \"odh-model-controller-7998c67dc4-sgxwt\" (UID: \"63fd10e0-82f9-444c-8302-2a31c8c1b98f\") " pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:37:48.527166 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.527133 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:37:48.798833 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.798762 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2dncm"] Sep 10 15:37:48.802313 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.802290 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:37:48.805203 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.805176 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 15:37:48.805516 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.805484 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-npf8b\"" Sep 10 15:37:48.811793 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.811736 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2dncm"] Sep 10 15:37:48.964098 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.964052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm798\" (UniqueName: \"kubernetes.io/projected/0e3d9454-6af8-4bea-9207-96ebd23b3bce-kube-api-access-nm798\") pod \"model-serving-api-54d89bd5fc-2dncm\" (UID: \"0e3d9454-6af8-4bea-9207-96ebd23b3bce\") " pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:37:48.964301 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:48.964154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3d9454-6af8-4bea-9207-96ebd23b3bce-tls-certs\") pod \"model-serving-api-54d89bd5fc-2dncm\" (UID: \"0e3d9454-6af8-4bea-9207-96ebd23b3bce\") " pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:37:49.064851 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.064762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3d9454-6af8-4bea-9207-96ebd23b3bce-tls-certs\") pod \"model-serving-api-54d89bd5fc-2dncm\" (UID: \"0e3d9454-6af8-4bea-9207-96ebd23b3bce\") " pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:37:49.065063 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.064863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm798\" (UniqueName: \"kubernetes.io/projected/0e3d9454-6af8-4bea-9207-96ebd23b3bce-kube-api-access-nm798\") pod \"model-serving-api-54d89bd5fc-2dncm\" (UID: \"0e3d9454-6af8-4bea-9207-96ebd23b3bce\") " pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:37:49.067258 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.067231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3d9454-6af8-4bea-9207-96ebd23b3bce-tls-certs\") pod \"model-serving-api-54d89bd5fc-2dncm\" (UID: \"0e3d9454-6af8-4bea-9207-96ebd23b3bce\") " pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:37:49.073235 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.073207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm798\" (UniqueName: \"kubernetes.io/projected/0e3d9454-6af8-4bea-9207-96ebd23b3bce-kube-api-access-nm798\") pod \"model-serving-api-54d89bd5fc-2dncm\" (UID: \"0e3d9454-6af8-4bea-9207-96ebd23b3bce\") " pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:37:49.116050 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.116014 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:37:49.143007 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.142880 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:37:49.143673 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.143208 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:37:49.143673 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.143267 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:37:49.341117 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.341011 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sgxwt"] Sep 10 15:37:49.344057 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:37:49.344024 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fd10e0_82f9_444c_8302_2a31c8c1b98f.slice/crio-454aaf4618c7b7540b79477434f79fe1ac00a5f0c88004271d5394fcd8e4e94c WatchSource:0}: Error finding container 454aaf4618c7b7540b79477434f79fe1ac00a5f0c88004271d5394fcd8e4e94c: Status 404 returned error can't find the container with id 454aaf4618c7b7540b79477434f79fe1ac00a5f0c88004271d5394fcd8e4e94c Sep 10 15:37:49.415697 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.415656 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-2dncm"] Sep 10 15:37:49.421930 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:37:49.421886 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3d9454_6af8_4bea_9207_96ebd23b3bce.slice/crio-f83a812c7411644b9dd93a533237c6ec2ee59c0d9881d8e733e121f1d738d48a WatchSource:0}: Error finding container f83a812c7411644b9dd93a533237c6ec2ee59c0d9881d8e733e121f1d738d48a: Status 404 returned error can't find the container with id f83a812c7411644b9dd93a533237c6ec2ee59c0d9881d8e733e121f1d738d48a Sep 10 15:37:49.565266 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.565223 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" event={"ID":"7c0b58fd-8f53-4a8a-ab2e-fd948778287d","Type":"ContainerStarted","Data":"1f70159aa7aaa31cbbe029bdff3f16f79a36a9e6e5bce4e23572776c9440d24f"} Sep 10 15:37:49.566746 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.566717 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" event={"ID":"0e3d9454-6af8-4bea-9207-96ebd23b3bce","Type":"ContainerStarted","Data":"f83a812c7411644b9dd93a533237c6ec2ee59c0d9881d8e733e121f1d738d48a"} Sep 10 15:37:49.568131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.568099 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" event={"ID":"63fd10e0-82f9-444c-8302-2a31c8c1b98f","Type":"ContainerStarted","Data":"454aaf4618c7b7540b79477434f79fe1ac00a5f0c88004271d5394fcd8e4e94c"} Sep 10 15:37:49.587525 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.587478 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" podStartSLOduration=1.513259391 podStartE2EDuration="5.587462348s" totalImagesPullingTime="4.074202957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:49.585804795 +0000 UTC m=+328.148113984" watchObservedRunningTime="2026-09-10 15:37:49.587462348 +0000 UTC m=+328.149771620" Sep 10 15:37:49.785659 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:49.785623 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:50.792643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:50.792603 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:51.579215 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:51.579179 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:51.580278 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:51.580245 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799c54rwr" Sep 10 15:37:54.596303 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:54.596265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" event={"ID":"0e3d9454-6af8-4bea-9207-96ebd23b3bce","Type":"ContainerStarted","Data":"52564505837eb492bebd1d7af1a264b7b1885325cf4193dfe215b1ee7f5e161f"} Sep 10 15:37:54.596777 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:54.596334 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:37:54.597852 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:54.597830 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" event={"ID":"63fd10e0-82f9-444c-8302-2a31c8c1b98f","Type":"ContainerStarted","Data":"e95df26b082b9dd324adfeafd3c0ff248a0a8209c6a54400ac195d4d0ff42c73"} Sep 10 15:37:54.597908 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:54.597893 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:37:54.619724 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:54.619675 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" podStartSLOduration=2.329877374 podStartE2EDuration="6.619659207s" totalImagesPullingTime="4.289781833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:54.617519238 +0000 UTC m=+333.179828429" watchObservedRunningTime="2026-09-10 15:37:54.619659207 +0000 UTC m=+333.181968454" Sep 10 15:37:54.644103 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:54.644029 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" podStartSLOduration=2.311827675 podStartE2EDuration="6.644006634s" totalImagesPullingTime="4.332178959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:54.643248612 +0000 UTC m=+333.205557803" watchObservedRunningTime="2026-09-10 15:37:54.644006634 +0000 UTC m=+333.206315820" Sep 10 15:37:56.168438 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.168398 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr"] Sep 10 15:37:56.171769 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.171745 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.174806 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.174785 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 15:37:56.174919 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.174830 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 15:37:56.174919 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.174841 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dr4sj\"" Sep 10 15:37:56.175078 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.175062 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 15:37:56.184686 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.184661 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr"] Sep 10 15:37:56.229134 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.229099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.229134 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.229140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.229328 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.229163 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.229328 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.229221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.229328 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.229245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7066d53c-5673-443c-bd2f-2dbdc3059bdd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.229328 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.229275 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7066d53c-5673-443c-bd2f-2dbdc3059bdd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.229472 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.229337 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcvb\" (UniqueName: \"kubernetes.io/projected/7066d53c-5673-443c-bd2f-2dbdc3059bdd-kube-api-access-kqcvb\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.330072 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.330038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcvb\" (UniqueName: \"kubernetes.io/projected/7066d53c-5673-443c-bd2f-2dbdc3059bdd-kube-api-access-kqcvb\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.330265 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.330083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.330265 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.330105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.330265 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.330127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.330265 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.330155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.330265 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.330194 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7066d53c-5673-443c-bd2f-2dbdc3059bdd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.330265 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.330218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7066d53c-5673-443c-bd2f-2dbdc3059bdd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.330995 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.330967 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.332636 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.332609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.332636 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.332631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.332811 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.332651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7066d53c-5673-443c-bd2f-2dbdc3059bdd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.332880 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.332834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7066d53c-5673-443c-bd2f-2dbdc3059bdd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.339286 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.339263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7066d53c-5673-443c-bd2f-2dbdc3059bdd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.339286 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.339278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcvb\" (UniqueName: \"kubernetes.io/projected/7066d53c-5673-443c-bd2f-2dbdc3059bdd-kube-api-access-kqcvb\") pod \"istiod-openshift-gateway-5677ccb49b-zbzjr\" (UID: \"7066d53c-5673-443c-bd2f-2dbdc3059bdd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.481351 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.481267 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:56.766339 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:56.766305 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr"] Sep 10 15:37:56.768772 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:37:56.768739 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7066d53c_5673_443c_bd2f_2dbdc3059bdd.slice/crio-f4028271559e74bd855315ffae90c81e1d46ea1ddbbe1fef54330f033b37d1b0 WatchSource:0}: Error finding container f4028271559e74bd855315ffae90c81e1d46ea1ddbbe1fef54330f033b37d1b0: Status 404 returned error can't find the container with id f4028271559e74bd855315ffae90c81e1d46ea1ddbbe1fef54330f033b37d1b0 Sep 10 15:37:57.614532 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:57.614488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" event={"ID":"7066d53c-5673-443c-bd2f-2dbdc3059bdd","Type":"ContainerStarted","Data":"f4028271559e74bd855315ffae90c81e1d46ea1ddbbe1fef54330f033b37d1b0"} Sep 10 15:37:58.556592 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:58.556560 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-556d846896-p5gdn" Sep 10 15:37:59.203399 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:59.203364 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:37:59.203682 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:59.203438 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:37:59.627993 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:59.627958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" event={"ID":"7066d53c-5673-443c-bd2f-2dbdc3059bdd","Type":"ContainerStarted","Data":"7f03d705bbd3ee0ffce77b5a11f02db4899a4c4c7a93dc61642febcd25ed89ba"} Sep 10 15:37:59.628191 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:59.628073 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:37:59.654039 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:37:59.653978 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" podStartSLOduration=1.222293542 podStartE2EDuration="3.653954777s" totalImagesPullingTime="2.431661235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:59.651028927 +0000 UTC m=+338.213338117" watchObservedRunningTime="2026-09-10 15:37:59.653954777 +0000 UTC m=+338.216264002" Sep 10 15:38:00.633779 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:38:00.633752 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zbzjr" Sep 10 15:38:05.605856 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:38:05.605825 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-sgxwt" Sep 10 15:38:05.607823 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:38:05.607801 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-2dncm" Sep 10 15:40:07.387379 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.387344 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58"] Sep 10 15:40:07.389934 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.389915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.392202 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.392180 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2htfb\"" Sep 10 15:40:07.403654 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.403627 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58"] Sep 10 15:40:07.469355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.469315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.469603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.469377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.469603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.469479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.469603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.469498 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.469603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.469526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.469603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.469546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.469772 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.469620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.469772 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.469662 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.469772 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.469682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrbsq\" (UniqueName: \"kubernetes.io/projected/60f42cf8-4bed-4d27-8d4e-766522ecb76c-kube-api-access-wrbsq\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570302 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570302 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrbsq\" (UniqueName: \"kubernetes.io/projected/60f42cf8-4bed-4d27-8d4e-766522ecb76c-kube-api-access-wrbsq\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570778 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570560 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570778 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570882 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570943 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570912 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.570994 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.570979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.571126 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.571105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.571170 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.571142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.572901 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.572879 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.573035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.573017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.578824 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.578797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60f42cf8-4bed-4d27-8d4e-766522ecb76c-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.579018 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.578995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrbsq\" (UniqueName: \"kubernetes.io/projected/60f42cf8-4bed-4d27-8d4e-766522ecb76c-kube-api-access-wrbsq\") pod \"maas-default-gateway-openshift-default-5ccb968c66-k7z58\" (UID: \"60f42cf8-4bed-4d27-8d4e-766522ecb76c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.700172 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.700083 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:07.975482 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.975376 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58"] Sep 10 15:40:07.978471 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:40:07.978440 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f42cf8_4bed_4d27_8d4e_766522ecb76c.slice/crio-5ada306dce5d5db732e76836367c9db1c442fb00c0134aa5646f07c5841e5775 WatchSource:0}: Error finding container 5ada306dce5d5db732e76836367c9db1c442fb00c0134aa5646f07c5841e5775: Status 404 returned error can't find the container with id 5ada306dce5d5db732e76836367c9db1c442fb00c0134aa5646f07c5841e5775 Sep 10 15:40:07.980559 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.980526 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:40:07.980664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.980589 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:40:07.980664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:07.980616 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:40:08.206500 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:08.206462 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" event={"ID":"60f42cf8-4bed-4d27-8d4e-766522ecb76c","Type":"ContainerStarted","Data":"8f603199c78ca18edc467e520b02d89890e36eccafe46377481fdbf578f96d9a"} Sep 10 15:40:08.206668 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:08.206527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" event={"ID":"60f42cf8-4bed-4d27-8d4e-766522ecb76c","Type":"ContainerStarted","Data":"5ada306dce5d5db732e76836367c9db1c442fb00c0134aa5646f07c5841e5775"} Sep 10 15:40:08.226411 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:08.226295 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" podStartSLOduration=1.226279241 podStartE2EDuration="1.226279241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:40:08.22434891 +0000 UTC m=+466.786658124" watchObservedRunningTime="2026-09-10 15:40:08.226279241 +0000 UTC m=+466.788588431" Sep 10 15:40:08.700290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:08.700251 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:09.706092 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:09.706063 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:10.214697 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:10.214666 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:40:10.215830 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:40:10.215807 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:41:00.435069 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.435038 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-569c478d95-cnfb2"] Sep 10 15:41:00.437736 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.437719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.441091 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.441063 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 15:41:00.441233 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.441073 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w22h2\"" Sep 10 15:41:00.441233 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.441072 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 15:41:00.441355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.441109 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 15:41:00.447526 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.447504 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569c478d95-cnfb2"] Sep 10 15:41:00.517631 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.517596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-metrics-certs\") pod \"maas-controller-569c478d95-cnfb2\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.517785 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.517655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-webhook-cert\") pod \"maas-controller-569c478d95-cnfb2\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.517785 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.517695 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg5f9\" (UniqueName: \"kubernetes.io/projected/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-kube-api-access-cg5f9\") pod \"maas-controller-569c478d95-cnfb2\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.618206 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.618172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-metrics-certs\") pod \"maas-controller-569c478d95-cnfb2\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.618379 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.618226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-webhook-cert\") pod \"maas-controller-569c478d95-cnfb2\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.618379 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.618257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5f9\" (UniqueName: \"kubernetes.io/projected/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-kube-api-access-cg5f9\") pod \"maas-controller-569c478d95-cnfb2\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.620750 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.620723 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-webhook-cert\") pod \"maas-controller-569c478d95-cnfb2\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.620848 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.620832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-metrics-certs\") pod \"maas-controller-569c478d95-cnfb2\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.626058 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.626026 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5f9\" (UniqueName: \"kubernetes.io/projected/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-kube-api-access-cg5f9\") pod \"maas-controller-569c478d95-cnfb2\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:00.749903 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:00.749826 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:01.019311 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:01.019289 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569c478d95-cnfb2"] Sep 10 15:41:01.021775 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:41:01.021743 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18070a4f_9f0a_4c1d_bd14_de83be91dc0a.slice/crio-aca24721cdeef506fe7670ab361a9169aad63457757304aeb7a34f93a5698f4a WatchSource:0}: Error finding container aca24721cdeef506fe7670ab361a9169aad63457757304aeb7a34f93a5698f4a: Status 404 returned error can't find the container with id aca24721cdeef506fe7670ab361a9169aad63457757304aeb7a34f93a5698f4a Sep 10 15:41:01.448704 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:01.448666 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-cnfb2" event={"ID":"18070a4f-9f0a-4c1d-bd14-de83be91dc0a","Type":"ContainerStarted","Data":"aca24721cdeef506fe7670ab361a9169aad63457757304aeb7a34f93a5698f4a"} Sep 10 15:41:04.463746 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:04.463713 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-cnfb2" event={"ID":"18070a4f-9f0a-4c1d-bd14-de83be91dc0a","Type":"ContainerStarted","Data":"b7d3c8ba3229c08f08605a87941099a953e3550fdff98d825e6f33d935105562"} Sep 10 15:41:04.464140 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:04.463851 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:04.490161 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:04.490105 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-569c478d95-cnfb2" podStartSLOduration=2.036915289 podStartE2EDuration="4.490090036s" totalImagesPullingTime="2.453174747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:04.486233897 +0000 UTC m=+523.048543088" watchObservedRunningTime="2026-09-10 15:41:04.490090036 +0000 UTC m=+523.052399239" Sep 10 15:41:15.470655 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:15.470571 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 15:41:40.530506 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.530470 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-qb9kv"] Sep 10 15:41:40.535725 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.535705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:40.538151 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.538113 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6cqbb\"" Sep 10 15:41:40.538273 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.538157 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 15:41:40.545802 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.545783 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-qb9kv"] Sep 10 15:41:40.568188 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.568156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kb2c\" (UniqueName: \"kubernetes.io/projected/fe46e4b2-e73b-4192-a133-bc065eb7f948-kube-api-access-9kb2c\") pod \"payload-processing-5b54b7fbb7-qb9kv\" (UID: \"fe46e4b2-e73b-4192-a133-bc065eb7f948\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:40.568297 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.568227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe46e4b2-e73b-4192-a133-bc065eb7f948-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-qb9kv\" (UID: \"fe46e4b2-e73b-4192-a133-bc065eb7f948\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:40.668817 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.668776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kb2c\" (UniqueName: \"kubernetes.io/projected/fe46e4b2-e73b-4192-a133-bc065eb7f948-kube-api-access-9kb2c\") pod \"payload-processing-5b54b7fbb7-qb9kv\" (UID: \"fe46e4b2-e73b-4192-a133-bc065eb7f948\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:40.668981 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.668855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe46e4b2-e73b-4192-a133-bc065eb7f948-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-qb9kv\" (UID: \"fe46e4b2-e73b-4192-a133-bc065eb7f948\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:40.669557 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.669537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe46e4b2-e73b-4192-a133-bc065eb7f948-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-qb9kv\" (UID: \"fe46e4b2-e73b-4192-a133-bc065eb7f948\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:40.677852 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.677818 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kb2c\" (UniqueName: \"kubernetes.io/projected/fe46e4b2-e73b-4192-a133-bc065eb7f948-kube-api-access-9kb2c\") pod \"payload-processing-5b54b7fbb7-qb9kv\" (UID: \"fe46e4b2-e73b-4192-a133-bc065eb7f948\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:40.847915 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:40.847828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:41.125583 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:41.125490 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-qb9kv"] Sep 10 15:41:41.142294 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:41:41.142235 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe46e4b2_e73b_4192_a133_bc065eb7f948.slice/crio-438c9297e4fccbe59cb4ab08f9cccb2d28cf6ba4908ac5c6c51d6e41d875c09f WatchSource:0}: Error finding container 438c9297e4fccbe59cb4ab08f9cccb2d28cf6ba4908ac5c6c51d6e41d875c09f: Status 404 returned error can't find the container with id 438c9297e4fccbe59cb4ab08f9cccb2d28cf6ba4908ac5c6c51d6e41d875c09f Sep 10 15:41:41.635971 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:41.635923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" event={"ID":"fe46e4b2-e73b-4192-a133-bc065eb7f948","Type":"ContainerStarted","Data":"438c9297e4fccbe59cb4ab08f9cccb2d28cf6ba4908ac5c6c51d6e41d875c09f"} Sep 10 15:41:44.653587 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:44.653552 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" event={"ID":"fe46e4b2-e73b-4192-a133-bc065eb7f948","Type":"ContainerStarted","Data":"2de17265693815995c87bd367bbe2014b9743db1cfd1c94bfb863513112643ff"} Sep 10 15:41:44.654017 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:44.653680 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:44.670837 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:44.670787 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" podStartSLOduration=1.849782592 podStartE2EDuration="4.670772567s" totalImagesPullingTime="2.820989975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:44.669268828 +0000 UTC m=+563.231578019" watchObservedRunningTime="2026-09-10 15:41:44.670772567 +0000 UTC m=+563.233081764" Sep 10 15:41:49.678597 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:49.678565 2580 generic.go:356] "Generic (PLEG): container finished" podID="60f42cf8-4bed-4d27-8d4e-766522ecb76c" containerID="8f603199c78ca18edc467e520b02d89890e36eccafe46377481fdbf578f96d9a" exitCode=0 Sep 10 15:41:49.679090 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:49.678643 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" event={"ID":"60f42cf8-4bed-4d27-8d4e-766522ecb76c","Type":"ContainerDied","Data":"8f603199c78ca18edc467e520b02d89890e36eccafe46377481fdbf578f96d9a"} Sep 10 15:41:49.679152 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:49.679137 2580 scope.go:117] "RemoveContainer" containerID="8f603199c78ca18edc467e520b02d89890e36eccafe46377481fdbf578f96d9a" Sep 10 15:41:49.680146 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:49.680120 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:41:49.680207 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:49.680187 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:41:49.680244 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:49.680216 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:41:50.685579 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:50.685541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" event={"ID":"60f42cf8-4bed-4d27-8d4e-766522ecb76c","Type":"ContainerStarted","Data":"93a601f9288be189b5c6aff0ae632599763c0f69121c7d34c6194dc9111d0321"} Sep 10 15:41:50.700565 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:50.700536 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:41:50.701544 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:50.701512 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-k7z58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 15:41:50.701656 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:50.701565 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" podUID="60f42cf8-4bed-4d27-8d4e-766522ecb76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 15:41:51.700585 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:51.700555 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-k7z58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 15:41:51.700964 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:51.700607 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" podUID="60f42cf8-4bed-4d27-8d4e-766522ecb76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 15:41:52.701491 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:52.701460 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-k7z58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 15:41:52.701892 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:52.701513 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" podUID="60f42cf8-4bed-4d27-8d4e-766522ecb76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 15:41:53.701173 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:53.701137 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-k7z58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 15:41:53.701343 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:53.701191 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" podUID="60f42cf8-4bed-4d27-8d4e-766522ecb76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 15:41:54.700613 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:54.700581 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-k7z58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Sep 10 15:41:54.700993 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:54.700639 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" podUID="60f42cf8-4bed-4d27-8d4e-766522ecb76c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Sep 10 15:41:55.215630 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:55.215581 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:41:55.660434 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:55.660401 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-qb9kv" Sep 10 15:41:55.724091 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:55.724051 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:41:55.725480 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:41:55.725450 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-k7z58" Sep 10 15:42:02.546860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.546818 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc"] Sep 10 15:42:02.550096 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.550068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.553277 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.553139 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 15:42:02.553277 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.553155 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 15:42:02.553277 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.553156 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ntw7b\"" Sep 10 15:42:02.553277 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.553218 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 15:42:02.557593 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.557574 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc"] Sep 10 15:42:02.674876 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.674839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.675041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.674893 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.675041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.674953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnq84\" (UniqueName: \"kubernetes.io/projected/6a95b701-9838-4162-a928-0d96dbb6f938-kube-api-access-gnq84\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.675041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.674975 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.675041 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.675024 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.675200 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.675052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a95b701-9838-4162-a928-0d96dbb6f938-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.776479 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.776444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnq84\" (UniqueName: \"kubernetes.io/projected/6a95b701-9838-4162-a928-0d96dbb6f938-kube-api-access-gnq84\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.776479 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.776483 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.776719 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.776510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.776719 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.776534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a95b701-9838-4162-a928-0d96dbb6f938-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.776719 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.776573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.776719 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.776637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.776981 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.776946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.777086 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.776983 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.777086 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.777002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.778830 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.778810 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a95b701-9838-4162-a928-0d96dbb6f938-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.779121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.779103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a95b701-9838-4162-a928-0d96dbb6f938-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.784075 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.784053 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnq84\" (UniqueName: \"kubernetes.io/projected/6a95b701-9838-4162-a928-0d96dbb6f938-kube-api-access-gnq84\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lsrhc\" (UID: \"6a95b701-9838-4162-a928-0d96dbb6f938\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:02.863889 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:02.863859 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:03.137608 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:03.137581 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc"] Sep 10 15:42:03.140148 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:42:03.140113 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a95b701_9838_4162_a928_0d96dbb6f938.slice/crio-42331268b4d2a86afbf8bb11f15a5155daef01dbc07156432304c6b1a3bfe959 WatchSource:0}: Error finding container 42331268b4d2a86afbf8bb11f15a5155daef01dbc07156432304c6b1a3bfe959: Status 404 returned error can't find the container with id 42331268b4d2a86afbf8bb11f15a5155daef01dbc07156432304c6b1a3bfe959 Sep 10 15:42:03.751746 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:03.751697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" event={"ID":"6a95b701-9838-4162-a928-0d96dbb6f938","Type":"ContainerStarted","Data":"42331268b4d2a86afbf8bb11f15a5155daef01dbc07156432304c6b1a3bfe959"} Sep 10 15:42:04.757704 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:04.757623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" event={"ID":"6a95b701-9838-4162-a928-0d96dbb6f938","Type":"ContainerStarted","Data":"288a1c28be7a9c09e753afee2b2499e283fa69b737ee5df39b089b46d233d442"} Sep 10 15:42:04.774014 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:04.773969 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" podStartSLOduration=1.513592187 podStartE2EDuration="2.773953564s" totalImagesPullingTime="1.260361377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:04.773025326 +0000 UTC m=+583.335334517" watchObservedRunningTime="2026-09-10 15:42:04.773953564 +0000 UTC m=+583.336262754" Sep 10 15:42:12.864256 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:12.864220 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:12.864698 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:12.864303 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:12.876448 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:12.876413 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:13.809798 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:13.809770 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lsrhc" Sep 10 15:42:15.844946 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.844912 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26"] Sep 10 15:42:15.848194 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.848176 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:15.850625 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.850598 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 15:42:15.856072 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.856051 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26"] Sep 10 15:42:15.989230 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.989196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:15.989230 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.989233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:15.989518 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.989271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:15.989518 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.989364 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:15.989518 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.989400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6kpn\" (UniqueName: \"kubernetes.io/projected/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-kube-api-access-l6kpn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:15.989518 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:15.989494 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.090569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.090532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.090758 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.090572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6kpn\" (UniqueName: \"kubernetes.io/projected/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-kube-api-access-l6kpn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.090758 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.090630 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.090866 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.090766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.090866 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.090809 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.090866 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.090860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.091131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.091111 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.091247 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.091167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.091247 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.091180 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.093037 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.093015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.093086 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.093024 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.097871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.097818 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6kpn\" (UniqueName: \"kubernetes.io/projected/994554d5-16b5-4cf4-a8ad-e0e94f16da9b-kube-api-access-l6kpn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qrv26\" (UID: \"994554d5-16b5-4cf4-a8ad-e0e94f16da9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.158910 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.158874 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:16.429646 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.429621 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26"] Sep 10 15:42:16.431679 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:42:16.431653 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994554d5_16b5_4cf4_a8ad_e0e94f16da9b.slice/crio-694f62ca857e38e318470af77234620e18193843abc16bac23fe30972a7260b3 WatchSource:0}: Error finding container 694f62ca857e38e318470af77234620e18193843abc16bac23fe30972a7260b3: Status 404 returned error can't find the container with id 694f62ca857e38e318470af77234620e18193843abc16bac23fe30972a7260b3 Sep 10 15:42:16.814501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.814400 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" event={"ID":"994554d5-16b5-4cf4-a8ad-e0e94f16da9b","Type":"ContainerStarted","Data":"f1120f2992fd7a22cfd5474ebfe549d188adaefbfe4fe86a19f83bcf400b1778"} Sep 10 15:42:16.814501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.814466 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" event={"ID":"994554d5-16b5-4cf4-a8ad-e0e94f16da9b","Type":"ContainerStarted","Data":"694f62ca857e38e318470af77234620e18193843abc16bac23fe30972a7260b3"} Sep 10 15:42:16.834872 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:16.834800 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" podStartSLOduration=1.660441013 podStartE2EDuration="1.834781707s" totalImagesPullingTime="174.340694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:16.83469469 +0000 UTC m=+595.397003886" watchObservedRunningTime="2026-09-10 15:42:16.834781707 +0000 UTC m=+595.397090897" Sep 10 15:42:21.968134 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:21.968103 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:42:21.968691 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:21.968412 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:42:26.159474 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:26.159414 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:26.159898 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:26.159497 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:26.171759 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:26.171732 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:26.873604 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:26.873573 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qrv26" Sep 10 15:42:57.427692 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.427613 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs"] Sep 10 15:42:57.431431 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.431397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.433758 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.433736 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 15:42:57.441841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.441816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs"] Sep 10 15:42:57.560276 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.560240 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.560489 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.560286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.560489 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.560312 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2bhw\" (UniqueName: \"kubernetes.io/projected/9c5060a8-b260-4aea-8710-7c53f612b8b0-kube-api-access-g2bhw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.560489 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.560378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.560489 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.560440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.560489 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.560458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5060a8-b260-4aea-8710-7c53f612b8b0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.661609 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.661577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.661609 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.661610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.661871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.661628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5060a8-b260-4aea-8710-7c53f612b8b0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.661871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.661687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.661871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.661716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.661871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.661742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2bhw\" (UniqueName: \"kubernetes.io/projected/9c5060a8-b260-4aea-8710-7c53f612b8b0-kube-api-access-g2bhw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.662137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.662112 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.662192 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.662135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.662192 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.662156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.663909 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.663889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5060a8-b260-4aea-8710-7c53f612b8b0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.664088 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.664065 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5060a8-b260-4aea-8710-7c53f612b8b0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.669725 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.669702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2bhw\" (UniqueName: \"kubernetes.io/projected/9c5060a8-b260-4aea-8710-7c53f612b8b0-kube-api-access-g2bhw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-c9tcs\" (UID: \"9c5060a8-b260-4aea-8710-7c53f612b8b0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:57.743117 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:57.743032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:42:58.014861 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:58.014837 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs"] Sep 10 15:42:58.016975 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:42:58.016949 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5060a8_b260_4aea_8710_7c53f612b8b0.slice/crio-3cb0e32846c7f0d4d82396430b446765773da26be46834bcc5e6632ff8603bf3 WatchSource:0}: Error finding container 3cb0e32846c7f0d4d82396430b446765773da26be46834bcc5e6632ff8603bf3: Status 404 returned error can't find the container with id 3cb0e32846c7f0d4d82396430b446765773da26be46834bcc5e6632ff8603bf3 Sep 10 15:42:58.018196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:58.018177 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:42:59.010218 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:59.010180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" event={"ID":"9c5060a8-b260-4aea-8710-7c53f612b8b0","Type":"ContainerStarted","Data":"e534e8515a382ed4d546f5309f43f18a450c3880ef5b1c0e41de34bdd9911891"} Sep 10 15:42:59.010631 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:59.010237 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" event={"ID":"9c5060a8-b260-4aea-8710-7c53f612b8b0","Type":"ContainerStarted","Data":"3cb0e32846c7f0d4d82396430b446765773da26be46834bcc5e6632ff8603bf3"} Sep 10 15:42:59.033271 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:42:59.032476 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" podStartSLOduration=1.8376916639999998 podStartE2EDuration="2.032451516s" totalImagesPullingTime="194.759852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:59.027564921 +0000 UTC m=+637.589874111" watchObservedRunningTime="2026-09-10 15:42:59.032451516 +0000 UTC m=+637.594760707" Sep 10 15:43:07.743656 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:43:07.743595 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:43:07.744032 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:43:07.743688 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:43:07.755952 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:43:07.755927 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:43:08.064457 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:43:08.064355 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-c9tcs" Sep 10 15:44:54.930857 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:54.930817 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5"] Sep 10 15:44:54.935130 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:54.935105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:54.937769 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:54.937745 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-80d47c-openshift-default-dockercfg-9bvz5\"" Sep 10 15:44:54.950090 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:54.950061 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5"] Sep 10 15:44:55.043336 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.043290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-data\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.043528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.043357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-istio-token\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.043528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.043396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-envoy\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.043528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.043443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-socket\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.043528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.043473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrr8h\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-kube-api-access-mrr8h\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.043528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.043501 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-certs\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.043723 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.043604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e5a659-ed15-4f90-8b16-076deddba214-istio-podinfo\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.043723 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.043630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e5a659-ed15-4f90-8b16-076deddba214-istiod-ca-cert\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.043723 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.043680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-credential-socket\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.144771 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.144732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-data\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.144968 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.144798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-istio-token\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.144968 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.144832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-envoy\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.144968 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.144858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-socket\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.144968 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.144882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrr8h\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-kube-api-access-mrr8h\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.144968 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.144909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-certs\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.144968 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.144945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e5a659-ed15-4f90-8b16-076deddba214-istio-podinfo\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.145263 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.144970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e5a659-ed15-4f90-8b16-076deddba214-istiod-ca-cert\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.145263 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.145015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-credential-socket\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.145468 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.145444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-socket\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.145559 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.145508 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-certs\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.145617 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.145557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-credential-socket\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.145855 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.145828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-data\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.145994 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.145828 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e5a659-ed15-4f90-8b16-076deddba214-istiod-ca-cert\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.147555 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.147533 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-envoy\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.147637 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.147594 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e5a659-ed15-4f90-8b16-076deddba214-istio-podinfo\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.152769 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.152743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-istio-token\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.152875 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.152859 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrr8h\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-kube-api-access-mrr8h\") pod \"e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.247935 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.247853 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:55.526006 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.525794 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5"] Sep 10 15:44:55.528758 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:44:55.528727 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e5a659_ed15_4f90_8b16_076deddba214.slice/crio-4bfde36df8c9e1cf66497aeb9c08c8ff40a1b8d4ac6d4446ee1a1c00c7a55238 WatchSource:0}: Error finding container 4bfde36df8c9e1cf66497aeb9c08c8ff40a1b8d4ac6d4446ee1a1c00c7a55238: Status 404 returned error can't find the container with id 4bfde36df8c9e1cf66497aeb9c08c8ff40a1b8d4ac6d4446ee1a1c00c7a55238 Sep 10 15:44:55.531099 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.531065 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:44:55.531176 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.531157 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:44:55.531223 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.531208 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:44:55.560748 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:55.560718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" event={"ID":"d0e5a659-ed15-4f90-8b16-076deddba214","Type":"ContainerStarted","Data":"4bfde36df8c9e1cf66497aeb9c08c8ff40a1b8d4ac6d4446ee1a1c00c7a55238"} Sep 10 15:44:56.567012 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:56.566980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" event={"ID":"d0e5a659-ed15-4f90-8b16-076deddba214","Type":"ContainerStarted","Data":"162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7"} Sep 10 15:44:56.588325 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:56.588278 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" podStartSLOduration=2.588260106 podStartE2EDuration="2.588260106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:56.586862782 +0000 UTC m=+755.149171972" watchObservedRunningTime="2026-09-10 15:44:56.588260106 +0000 UTC m=+755.150569295" Sep 10 15:44:57.248143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:57.248108 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:57.253087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:57.253059 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:57.578162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:57.578076 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:44:57.578967 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:44:57.578949 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:45:00.137655 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:00.137620 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd"] Sep 10 15:45:00.141181 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:00.141163 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd" Sep 10 15:45:00.143508 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:00.143486 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:45:00.147985 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:00.147959 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd"] Sep 10 15:45:00.153913 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:00.153892 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd" Sep 10 15:45:00.428221 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:00.428200 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd"] Sep 10 15:45:00.430539 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:45:00.430511 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec595964_a1b9_4d9b_accd_23b5cedf3773.slice/crio-69a3d9f2096484efccc4dda43cf8b6997324372b793adfecc4a615d59a6d37a6 WatchSource:0}: Error finding container 69a3d9f2096484efccc4dda43cf8b6997324372b793adfecc4a615d59a6d37a6: Status 404 returned error can't find the container with id 69a3d9f2096484efccc4dda43cf8b6997324372b793adfecc4a615d59a6d37a6 Sep 10 15:45:00.595500 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:00.595462 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd" event={"ID":"ec595964-a1b9-4d9b-accd-23b5cedf3773","Type":"ContainerStarted","Data":"69a3d9f2096484efccc4dda43cf8b6997324372b793adfecc4a615d59a6d37a6"} Sep 10 15:45:03.014337 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.014253 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx"] Sep 10 15:45:03.019386 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.019361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.021926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.021899 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7083b0df-openshift-default-dockercfg-n2kn2\"" Sep 10 15:45:03.028160 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.028129 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx"] Sep 10 15:45:03.126672 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.126635 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-certs\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.126672 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.126672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-data\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.126890 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.126706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhw5p\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-kube-api-access-zhw5p\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.126890 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.126746 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-podinfo\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.126890 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.126776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-socket\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.126890 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.126792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-envoy\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.126890 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.126809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-token\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.126890 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.126880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istiod-ca-cert\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.127095 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.126960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-credential-socket\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228169 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-credential-socket\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228352 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-certs\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228352 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-data\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228502 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhw5p\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-kube-api-access-zhw5p\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228561 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228501 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-podinfo\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228561 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228531 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-credential-socket\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228674 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228653 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-certs\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228732 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-socket\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228732 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-data\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228732 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-envoy\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228900 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-token\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228900 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istiod-ca-cert\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.228900 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.228889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-socket\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.229412 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.229390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istiod-ca-cert\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.230894 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.230871 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-envoy\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.231149 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.231126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-podinfo\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.236844 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.236815 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-token\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.237021 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.237004 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhw5p\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-kube-api-access-zhw5p\") pod \"e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.332266 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.332185 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:03.612888 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.612853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd" event={"ID":"ec595964-a1b9-4d9b-accd-23b5cedf3773","Type":"ContainerStarted","Data":"5e830da814200822b00610f294d0733f75c81d071ef5d89d9261e0e45cc250cf"} Sep 10 15:45:03.616250 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.616220 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx"] Sep 10 15:45:03.619262 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:45:03.619235 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9668fbe_fd8d_4d60_afde_5a7459e0f9ff.slice/crio-5e8b1753a5e655c49c605b8917b16c5267f95ff36b8e031d8fb3143dadff0a64 WatchSource:0}: Error finding container 5e8b1753a5e655c49c605b8917b16c5267f95ff36b8e031d8fb3143dadff0a64: Status 404 returned error can't find the container with id 5e8b1753a5e655c49c605b8917b16c5267f95ff36b8e031d8fb3143dadff0a64 Sep 10 15:45:03.621786 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.621757 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:03.621857 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.621824 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:03.621894 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.621855 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:03.626732 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:03.626685 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd" podStartSLOduration=1.3334250060000001 podStartE2EDuration="3.626670714s" totalImagesPullingTime="2.293245708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:03.626015439 +0000 UTC m=+762.188324626" watchObservedRunningTime="2026-09-10 15:45:03.626670714 +0000 UTC m=+762.188979925" Sep 10 15:45:04.625887 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:04.620601 2580 generic.go:356] "Generic (PLEG): container finished" podID="ec595964-a1b9-4d9b-accd-23b5cedf3773" containerID="5e830da814200822b00610f294d0733f75c81d071ef5d89d9261e0e45cc250cf" exitCode=0 Sep 10 15:45:04.625887 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:04.620749 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd" event={"ID":"ec595964-a1b9-4d9b-accd-23b5cedf3773","Type":"ContainerDied","Data":"5e830da814200822b00610f294d0733f75c81d071ef5d89d9261e0e45cc250cf"} Sep 10 15:45:04.628030 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:04.628001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" event={"ID":"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff","Type":"ContainerStarted","Data":"0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b"} Sep 10 15:45:04.628187 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:04.628055 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" event={"ID":"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff","Type":"ContainerStarted","Data":"5e8b1753a5e655c49c605b8917b16c5267f95ff36b8e031d8fb3143dadff0a64"} Sep 10 15:45:04.658718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:04.658673 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" podStartSLOduration=2.65865512 podStartE2EDuration="2.65865512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:04.657272305 +0000 UTC m=+763.219581496" watchObservedRunningTime="2026-09-10 15:45:04.65865512 +0000 UTC m=+763.220964310" Sep 10 15:45:05.332691 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:05.332651 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:05.337528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:05.337499 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:05.633563 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:05.633124 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:05.633942 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:05.633887 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:05.779612 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:05.779581 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd" Sep 10 15:45:06.638372 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:06.638336 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd" Sep 10 15:45:06.638802 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:06.638370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-fcvrd" event={"ID":"ec595964-a1b9-4d9b-accd-23b5cedf3773","Type":"ContainerDied","Data":"69a3d9f2096484efccc4dda43cf8b6997324372b793adfecc4a615d59a6d37a6"} Sep 10 15:45:06.638802 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:06.638415 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69a3d9f2096484efccc4dda43cf8b6997324372b793adfecc4a615d59a6d37a6" Sep 10 15:45:08.009464 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.009410 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d"] Sep 10 15:45:08.009869 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.009853 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec595964-a1b9-4d9b-accd-23b5cedf3773" containerName="cleanup" Sep 10 15:45:08.009912 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.009871 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec595964-a1b9-4d9b-accd-23b5cedf3773" containerName="cleanup" Sep 10 15:45:08.009967 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.009957 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec595964-a1b9-4d9b-accd-23b5cedf3773" containerName="cleanup" Sep 10 15:45:08.012979 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.012964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:08.015087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.015065 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7083b0df-dockercfg-b7whs\"" Sep 10 15:45:08.015214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.015173 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7083b0df\"" Sep 10 15:45:08.030026 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.029999 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d"] Sep 10 15:45:08.180830 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.180787 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abcdc54-cb44-4af0-8c87-da47c9358c2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d\" (UID: \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:08.181026 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.180870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsbpn\" (UniqueName: \"kubernetes.io/projected/7abcdc54-cb44-4af0-8c87-da47c9358c2d-kube-api-access-qsbpn\") pod \"payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d\" (UID: \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:08.282255 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.282171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abcdc54-cb44-4af0-8c87-da47c9358c2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d\" (UID: \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:08.282255 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.282230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsbpn\" (UniqueName: \"kubernetes.io/projected/7abcdc54-cb44-4af0-8c87-da47c9358c2d-kube-api-access-qsbpn\") pod \"payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d\" (UID: \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:08.282927 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.282905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abcdc54-cb44-4af0-8c87-da47c9358c2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d\" (UID: \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:08.290230 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.290207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsbpn\" (UniqueName: \"kubernetes.io/projected/7abcdc54-cb44-4af0-8c87-da47c9358c2d-kube-api-access-qsbpn\") pod \"payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d\" (UID: \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:08.327444 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.327105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:08.632661 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.627276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d"] Sep 10 15:45:08.643298 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:45:08.637338 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abcdc54_cb44_4af0_8c87_da47c9358c2d.slice/crio-4e1189c56fe9e00e9f6cb8b3c8333822e59208bf844d2dd7ff98b1a613a52053 WatchSource:0}: Error finding container 4e1189c56fe9e00e9f6cb8b3c8333822e59208bf844d2dd7ff98b1a613a52053: Status 404 returned error can't find the container with id 4e1189c56fe9e00e9f6cb8b3c8333822e59208bf844d2dd7ff98b1a613a52053 Sep 10 15:45:08.666842 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:08.666805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" event={"ID":"7abcdc54-cb44-4af0-8c87-da47c9358c2d","Type":"ContainerStarted","Data":"4e1189c56fe9e00e9f6cb8b3c8333822e59208bf844d2dd7ff98b1a613a52053"} Sep 10 15:45:09.673477 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:09.673440 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" event={"ID":"7abcdc54-cb44-4af0-8c87-da47c9358c2d","Type":"ContainerStarted","Data":"db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495"} Sep 10 15:45:09.673848 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:09.673519 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:09.689351 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:09.689301 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" podStartSLOduration=2.6892869470000003 podStartE2EDuration="2.689286947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:09.688070344 +0000 UTC m=+768.250379533" watchObservedRunningTime="2026-09-10 15:45:09.689286947 +0000 UTC m=+768.251596136" Sep 10 15:45:11.461869 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.461836 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z"] Sep 10 15:45:11.465939 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.465916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.468571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.468547 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-d5048f-openshift-default-dockercfg-p9ws6\"" Sep 10 15:45:11.478072 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.478049 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z"] Sep 10 15:45:11.631616 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.631576 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d988b-e54c-4552-8fec-f9890599c92f-istiod-ca-cert\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.631887 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.631863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-istio-token\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.632022 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.632000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d988b-e54c-4552-8fec-f9890599c92f-istio-podinfo\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.632172 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.632156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-data\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.632351 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.632310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-socket\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.632484 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.632469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-credential-socket\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.632720 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.632697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-certs\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.632909 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.632892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhzx\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-kube-api-access-cfhzx\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.633047 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.633032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-envoy\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.734801 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.734693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhzx\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-kube-api-access-cfhzx\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.734801 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.734773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-envoy\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.735035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.734808 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d988b-e54c-4552-8fec-f9890599c92f-istiod-ca-cert\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.735035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.734841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-istio-token\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.735035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.734871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d988b-e54c-4552-8fec-f9890599c92f-istio-podinfo\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.735035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.734918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-data\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.735035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.734955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-socket\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.735035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.734977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-credential-socket\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.735035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.735012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-certs\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.735619 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.735473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-certs\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.735619 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.735490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-data\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.736878 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.735793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-credential-socket\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.736878 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.736773 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d988b-e54c-4552-8fec-f9890599c92f-istiod-ca-cert\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.737381 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.737346 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-socket\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.739084 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.739042 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d988b-e54c-4552-8fec-f9890599c92f-istio-podinfo\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.739352 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.739333 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-envoy\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.746445 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.743852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-istio-token\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.749452 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.748064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhzx\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-kube-api-access-cfhzx\") pod \"e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:11.778663 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:11.778630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:12.078083 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:12.077935 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z"] Sep 10 15:45:12.080382 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:45:12.080343 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73d988b_e54c_4552_8fec_f9890599c92f.slice/crio-2349e4bc5df3773e5ee5a333b96ac2aa7ee286bf38341ca211f675b5e4fd5e90 WatchSource:0}: Error finding container 2349e4bc5df3773e5ee5a333b96ac2aa7ee286bf38341ca211f675b5e4fd5e90: Status 404 returned error can't find the container with id 2349e4bc5df3773e5ee5a333b96ac2aa7ee286bf38341ca211f675b5e4fd5e90 Sep 10 15:45:12.082925 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:12.082885 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:12.083014 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:12.082977 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:12.083054 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:12.083021 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:12.690491 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:12.690455 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" event={"ID":"d73d988b-e54c-4552-8fec-f9890599c92f","Type":"ContainerStarted","Data":"cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486"} Sep 10 15:45:12.690872 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:12.690532 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" event={"ID":"d73d988b-e54c-4552-8fec-f9890599c92f","Type":"ContainerStarted","Data":"2349e4bc5df3773e5ee5a333b96ac2aa7ee286bf38341ca211f675b5e4fd5e90"} Sep 10 15:45:12.712265 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:12.712211 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" podStartSLOduration=1.7121928560000002 podStartE2EDuration="1.712192856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:12.709486512 +0000 UTC m=+771.271795703" watchObservedRunningTime="2026-09-10 15:45:12.712192856 +0000 UTC m=+771.274502048" Sep 10 15:45:12.779109 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:12.779074 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:13.784733 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:13.784702 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:14.701372 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:14.701337 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:14.702405 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:14.702382 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:17.732715 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:17.732677 2580 generic.go:356] "Generic (PLEG): container finished" podID="d0e5a659-ed15-4f90-8b16-076deddba214" containerID="162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7" exitCode=0 Sep 10 15:45:17.733506 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:17.733353 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" event={"ID":"d0e5a659-ed15-4f90-8b16-076deddba214","Type":"ContainerDied","Data":"162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7"} Sep 10 15:45:17.734200 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:17.734184 2580 scope.go:117] "RemoveContainer" containerID="162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7" Sep 10 15:45:17.736436 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:17.736069 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:17.736436 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:17.736162 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:17.736436 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:17.736204 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:18.744794 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:18.744752 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" event={"ID":"d0e5a659-ed15-4f90-8b16-076deddba214","Type":"ContainerStarted","Data":"af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44"} Sep 10 15:45:19.248492 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:19.248446 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:45:19.248934 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:19.248911 2580 patch_prober.go:28] interesting pod/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:45:19.249126 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:19.249090 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:45:20.249124 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:20.249082 2580 patch_prober.go:28] interesting pod/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:45:20.249727 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:20.249693 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:45:20.680550 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:20.680515 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:21.248457 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:21.248392 2580 patch_prober.go:28] interesting pod/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:45:21.248753 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:21.248720 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:45:22.248282 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:22.248243 2580 patch_prober.go:28] interesting pod/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:45:22.248777 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:22.248321 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:45:23.249030 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:23.248984 2580 patch_prober.go:28] interesting pod/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Sep 10 15:45:23.249403 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:23.249068 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Sep 10 15:45:24.119128 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.119089 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2"] Sep 10 15:45:24.134448 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.124650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.134448 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.127354 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8a850d-openshift-default-dockercfg-rkr48\"" Sep 10 15:45:24.136685 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.136658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2"] Sep 10 15:45:24.214694 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.214653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds4rz\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-kube-api-access-ds4rz\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.214873 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.214769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-socket\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.214873 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.214823 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-certs\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.214956 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.214897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-data\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.214996 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.214985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-podinfo\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.215046 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.215031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-credential-socket\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.215094 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.215063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istiod-ca-cert\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.215130 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.215091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-token\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.215164 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.215135 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-envoy\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.316412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-token\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.316500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-envoy\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.316537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds4rz\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-kube-api-access-ds4rz\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.316999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-socket\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.317043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-socket\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.317141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-certs\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.317214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-data\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.317315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-podinfo\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.317359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-credential-socket\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.317380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.317387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istiod-ca-cert\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.318553 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.318390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-credential-socket\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.318553 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.318406 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-data\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.318553 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.318504 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-certs\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.319646 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.319626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istiod-ca-cert\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.320782 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.320753 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-envoy\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.321956 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.321913 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-podinfo\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.328929 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.328899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-token\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.330988 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.330958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds4rz\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-kube-api-access-ds4rz\") pod \"e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.447518 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.447434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:24.620854 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.620237 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:45:24.621052 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.621038 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:45:24.622877 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.622822 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:45:24.777847 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.777816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2"] Sep 10 15:45:24.780841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.780796 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:24.780994 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.780890 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:24.780994 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.780933 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:24.805184 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:24.805152 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" event={"ID":"103dddaa-4e90-4d2c-9c93-184aee3eeccd","Type":"ContainerStarted","Data":"d5e3f7554e61cb2624a8119ef21b50fc58653557b45003e9b60edd3515a05baa"} Sep 10 15:45:25.812173 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:25.812139 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" event={"ID":"103dddaa-4e90-4d2c-9c93-184aee3eeccd","Type":"ContainerStarted","Data":"bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae"} Sep 10 15:45:25.835445 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:25.835363 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" podStartSLOduration=1.8353424600000001 podStartE2EDuration="1.83534246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:25.83169352 +0000 UTC m=+784.394002712" watchObservedRunningTime="2026-09-10 15:45:25.83534246 +0000 UTC m=+784.397651653" Sep 10 15:45:26.448189 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.448151 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:26.453294 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.453272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:26.822761 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.822652 2580 generic.go:356] "Generic (PLEG): container finished" podID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerID="0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b" exitCode=0 Sep 10 15:45:26.824049 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.824017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" event={"ID":"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff","Type":"ContainerDied","Data":"0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b"} Sep 10 15:45:26.824179 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.824102 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:26.824602 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.824575 2580 scope.go:117] "RemoveContainer" containerID="0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b" Sep 10 15:45:26.826213 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.826179 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:26.826303 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.826264 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:26.826368 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.826311 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:26.841645 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:26.841628 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:27.834506 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:27.834468 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" event={"ID":"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff","Type":"ContainerStarted","Data":"d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9"} Sep 10 15:45:28.333353 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:28.333310 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:28.333590 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:28.333458 2580 patch_prober.go:28] interesting pod/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:45:28.333590 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:28.333507 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:45:29.333551 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.333511 2580 patch_prober.go:28] interesting pod/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:45:29.334032 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.333590 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:45:29.796747 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.796716 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx"] Sep 10 15:45:29.799652 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.799636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:29.802782 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.802765 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8a850d-dockercfg-8mp4h\"" Sep 10 15:45:29.802944 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.802932 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8a850d\"" Sep 10 15:45:29.810255 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.810157 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx"] Sep 10 15:45:29.888975 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.888933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx6nw\" (UniqueName: \"kubernetes.io/projected/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-kube-api-access-kx6nw\") pod \"payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx\" (UID: \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:29.889163 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.888998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx\" (UID: \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:29.990698 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.990655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx6nw\" (UniqueName: \"kubernetes.io/projected/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-kube-api-access-kx6nw\") pod \"payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx\" (UID: \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:29.991047 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.990718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx\" (UID: \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:29.991459 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.991436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx\" (UID: \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:29.998176 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:29.998158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx6nw\" (UniqueName: \"kubernetes.io/projected/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-kube-api-access-kx6nw\") pod \"payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx\" (UID: \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:30.112598 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:30.112563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:30.333452 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:30.332800 2580 patch_prober.go:28] interesting pod/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:45:30.333452 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:30.332864 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:45:30.431497 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:30.428321 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx"] Sep 10 15:45:30.446569 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:45:30.446541 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f28eb6a_aaa4_47b1_a8a1_f1c6774f1e24.slice/crio-7b69b2e3bd5731f3ac5791f2b4b1e6910b97567d7c328a174eaebc6b1294c2b0 WatchSource:0}: Error finding container 7b69b2e3bd5731f3ac5791f2b4b1e6910b97567d7c328a174eaebc6b1294c2b0: Status 404 returned error can't find the container with id 7b69b2e3bd5731f3ac5791f2b4b1e6910b97567d7c328a174eaebc6b1294c2b0 Sep 10 15:45:30.863727 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:30.863693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" event={"ID":"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24","Type":"ContainerStarted","Data":"b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b"} Sep 10 15:45:30.863884 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:30.863764 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:30.863884 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:30.863793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" event={"ID":"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24","Type":"ContainerStarted","Data":"7b69b2e3bd5731f3ac5791f2b4b1e6910b97567d7c328a174eaebc6b1294c2b0"} Sep 10 15:45:30.881842 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:30.881788 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" podStartSLOduration=1.8817665940000001 podStartE2EDuration="1.881766594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:30.879441237 +0000 UTC m=+789.441750419" watchObservedRunningTime="2026-09-10 15:45:30.881766594 +0000 UTC m=+789.444075785" Sep 10 15:45:31.336940 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:31.333464 2580 patch_prober.go:28] interesting pod/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:45:31.336940 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:31.333540 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:45:32.337448 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:32.332787 2580 patch_prober.go:28] interesting pod/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 15:45:32.337448 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:32.332849 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 15:45:33.373703 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:33.373667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:33.374436 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:33.374391 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:33.375305 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:33.375284 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:34.890331 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:34.890297 2580 generic.go:356] "Generic (PLEG): container finished" podID="d73d988b-e54c-4552-8fec-f9890599c92f" containerID="cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486" exitCode=0 Sep 10 15:45:34.890794 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:34.890371 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" event={"ID":"d73d988b-e54c-4552-8fec-f9890599c92f","Type":"ContainerDied","Data":"cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486"} Sep 10 15:45:34.890973 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:34.890957 2580 scope.go:117] "RemoveContainer" containerID="cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486" Sep 10 15:45:34.892071 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:34.892038 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:34.892135 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:34.892108 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:34.892176 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:34.892137 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:35.898267 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:35.898229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" event={"ID":"d73d988b-e54c-4552-8fec-f9890599c92f","Type":"ContainerStarted","Data":"b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10"} Sep 10 15:45:36.658709 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.658669 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf"] Sep 10 15:45:36.662364 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.662337 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" Sep 10 15:45:36.665462 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.665305 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:45:36.665462 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.665325 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:45:36.669290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.669258 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf"] Sep 10 15:45:36.766057 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.766021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035d3495-525f-4d05-8128-f2b1967de117-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf\" (UID: \"035d3495-525f-4d05-8128-f2b1967de117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" Sep 10 15:45:36.779801 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.779770 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:36.780085 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.780051 2580 patch_prober.go:28] interesting pod/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:45:36.780186 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.780111 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:45:36.867431 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.867372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035d3495-525f-4d05-8128-f2b1967de117-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf\" (UID: \"035d3495-525f-4d05-8128-f2b1967de117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" Sep 10 15:45:36.868017 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.867992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035d3495-525f-4d05-8128-f2b1967de117-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf\" (UID: \"035d3495-525f-4d05-8128-f2b1967de117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" Sep 10 15:45:36.982323 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:36.982238 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" Sep 10 15:45:37.276160 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:37.276129 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf"] Sep 10 15:45:37.277933 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:45:37.277906 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035d3495_525f_4d05_8128_f2b1967de117.slice/crio-8e363a308bcebf08b604885893131bfcf94e16d49bf2638e3f1b2058f6641382 WatchSource:0}: Error finding container 8e363a308bcebf08b604885893131bfcf94e16d49bf2638e3f1b2058f6641382: Status 404 returned error can't find the container with id 8e363a308bcebf08b604885893131bfcf94e16d49bf2638e3f1b2058f6641382 Sep 10 15:45:37.779864 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:37.779792 2580 patch_prober.go:28] interesting pod/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:45:37.779864 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:37.779846 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:45:37.911378 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:37.911336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" event={"ID":"035d3495-525f-4d05-8128-f2b1967de117","Type":"ContainerStarted","Data":"6954ed3002a2234334c98fdce003d901f272a7f4c48d29f24ccd23ae6f224ed4"} Sep 10 15:45:37.911572 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:37.911408 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" event={"ID":"035d3495-525f-4d05-8128-f2b1967de117","Type":"ContainerStarted","Data":"8e363a308bcebf08b604885893131bfcf94e16d49bf2638e3f1b2058f6641382"} Sep 10 15:45:37.928572 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:37.928495 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" podStartSLOduration=1.928475096 podStartE2EDuration="1.928475096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:37.926645976 +0000 UTC m=+796.488955167" watchObservedRunningTime="2026-09-10 15:45:37.928475096 +0000 UTC m=+796.490784289" Sep 10 15:45:38.779603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:38.779131 2580 patch_prober.go:28] interesting pod/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:45:38.779603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:38.779196 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:45:38.922130 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:38.922093 2580 generic.go:356] "Generic (PLEG): container finished" podID="035d3495-525f-4d05-8128-f2b1967de117" containerID="6954ed3002a2234334c98fdce003d901f272a7f4c48d29f24ccd23ae6f224ed4" exitCode=0 Sep 10 15:45:38.922318 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:38.922167 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" event={"ID":"035d3495-525f-4d05-8128-f2b1967de117","Type":"ContainerDied","Data":"6954ed3002a2234334c98fdce003d901f272a7f4c48d29f24ccd23ae6f224ed4"} Sep 10 15:45:39.779455 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:39.779401 2580 patch_prober.go:28] interesting pod/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:45:39.779762 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:39.779722 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:45:40.193923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:40.193902 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" Sep 10 15:45:40.308227 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:40.308196 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035d3495-525f-4d05-8128-f2b1967de117-maas-api-service-ca\") pod \"035d3495-525f-4d05-8128-f2b1967de117\" (UID: \"035d3495-525f-4d05-8128-f2b1967de117\") " Sep 10 15:45:40.308557 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:40.308537 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035d3495-525f-4d05-8128-f2b1967de117-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "035d3495-525f-4d05-8128-f2b1967de117" (UID: "035d3495-525f-4d05-8128-f2b1967de117"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:40.409959 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:40.409923 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/035d3495-525f-4d05-8128-f2b1967de117-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:40.779570 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:40.779528 2580 patch_prober.go:28] interesting pod/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Sep 10 15:45:40.779769 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:40.779598 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Sep 10 15:45:40.940862 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:40.940819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" event={"ID":"035d3495-525f-4d05-8128-f2b1967de117","Type":"ContainerDied","Data":"8e363a308bcebf08b604885893131bfcf94e16d49bf2638e3f1b2058f6641382"} Sep 10 15:45:40.941065 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:40.940885 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e363a308bcebf08b604885893131bfcf94e16d49bf2638e3f1b2058f6641382" Sep 10 15:45:40.941065 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:40.940951 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf" Sep 10 15:45:41.782841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:41.782813 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:41.783215 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:41.783095 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:41.783874 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:41.783857 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:45:41.871509 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:41.871483 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:45:47.979608 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:47.979575 2580 generic.go:356] "Generic (PLEG): container finished" podID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerID="bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae" exitCode=0 Sep 10 15:45:47.980128 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:47.979655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" event={"ID":"103dddaa-4e90-4d2c-9c93-184aee3eeccd","Type":"ContainerDied","Data":"bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae"} Sep 10 15:45:47.980128 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:47.980110 2580 scope.go:117] "RemoveContainer" containerID="bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae" Sep 10 15:45:47.981254 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:47.981227 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:47.981365 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:47.981291 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:47.981365 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:47.981324 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:45:48.304071 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.302533 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d"] Sep 10 15:45:48.304071 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.302772 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" podUID="7abcdc54-cb44-4af0-8c87-da47c9358c2d" containerName="payload-pre-processing" containerID="cri-o://db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495" gracePeriod=30 Sep 10 15:45:48.463454 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.463404 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf"] Sep 10 15:45:48.467211 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.467187 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7083b0df-lzvdf"] Sep 10 15:45:48.601710 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.601659 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx"] Sep 10 15:45:48.602126 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.602108 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:48.602325 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.602205 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" containerID="cri-o://d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9" gracePeriod=30 Sep 10 15:45:48.707505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.706723 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsbpn\" (UniqueName: \"kubernetes.io/projected/7abcdc54-cb44-4af0-8c87-da47c9358c2d-kube-api-access-qsbpn\") pod \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\" (UID: \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\") " Sep 10 15:45:48.707505 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.706823 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abcdc54-cb44-4af0-8c87-da47c9358c2d-plugins-config-volume\") pod \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\" (UID: \"7abcdc54-cb44-4af0-8c87-da47c9358c2d\") " Sep 10 15:45:48.707959 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.707911 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abcdc54-cb44-4af0-8c87-da47c9358c2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7abcdc54-cb44-4af0-8c87-da47c9358c2d" (UID: "7abcdc54-cb44-4af0-8c87-da47c9358c2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:48.723607 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.723563 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abcdc54-cb44-4af0-8c87-da47c9358c2d-kube-api-access-qsbpn" (OuterVolumeSpecName: "kube-api-access-qsbpn") pod "7abcdc54-cb44-4af0-8c87-da47c9358c2d" (UID: "7abcdc54-cb44-4af0-8c87-da47c9358c2d"). InnerVolumeSpecName "kube-api-access-qsbpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:45:48.808230 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.808141 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsbpn\" (UniqueName: \"kubernetes.io/projected/7abcdc54-cb44-4af0-8c87-da47c9358c2d-kube-api-access-qsbpn\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:48.808230 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.808172 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abcdc54-cb44-4af0-8c87-da47c9358c2d-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:48.989365 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.989327 2580 generic.go:356] "Generic (PLEG): container finished" podID="7abcdc54-cb44-4af0-8c87-da47c9358c2d" containerID="db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495" exitCode=0 Sep 10 15:45:48.989816 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.989370 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" Sep 10 15:45:48.989816 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.989409 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" event={"ID":"7abcdc54-cb44-4af0-8c87-da47c9358c2d","Type":"ContainerDied","Data":"db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495"} Sep 10 15:45:48.989816 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.989473 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d" event={"ID":"7abcdc54-cb44-4af0-8c87-da47c9358c2d","Type":"ContainerDied","Data":"4e1189c56fe9e00e9f6cb8b3c8333822e59208bf844d2dd7ff98b1a613a52053"} Sep 10 15:45:48.989816 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.989496 2580 scope.go:117] "RemoveContainer" containerID="db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495" Sep 10 15:45:48.991593 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.991573 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" event={"ID":"103dddaa-4e90-4d2c-9c93-184aee3eeccd","Type":"ContainerStarted","Data":"bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d"} Sep 10 15:45:48.998930 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.998916 2580 scope.go:117] "RemoveContainer" containerID="db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495" Sep 10 15:45:48.999206 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:45:48.999186 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495\": container with ID starting with db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495 not found: ID does not exist" containerID="db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495" Sep 10 15:45:48.999288 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:48.999213 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495"} err="failed to get container status \"db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495\": rpc error: code = NotFound desc = could not find container \"db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495\": container with ID starting with db82b8aabd22261287ede7101f990def67a26e06b2c3d033c9fcef9021331495 not found: ID does not exist" Sep 10 15:45:49.039340 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:49.039295 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d"] Sep 10 15:45:49.042854 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:49.042813 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7083b0df-86f7646c5d-hrt6d"] Sep 10 15:45:49.447996 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:49.447958 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:49.448189 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:49.448150 2580 patch_prober.go:28] interesting pod/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:45:49.448240 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:49.448210 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:45:50.032323 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:50.032282 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="035d3495-525f-4d05-8128-f2b1967de117" path="/var/lib/kubelet/pods/035d3495-525f-4d05-8128-f2b1967de117/volumes" Sep 10 15:45:50.032901 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:50.032874 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abcdc54-cb44-4af0-8c87-da47c9358c2d" path="/var/lib/kubelet/pods/7abcdc54-cb44-4af0-8c87-da47c9358c2d/volumes" Sep 10 15:45:50.448579 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:50.448542 2580 patch_prober.go:28] interesting pod/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:45:50.448787 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:50.448603 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:45:51.448461 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:51.448408 2580 patch_prober.go:28] interesting pod/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:45:51.448929 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:51.448494 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:45:52.448340 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:52.448304 2580 patch_prober.go:28] interesting pod/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:45:52.448545 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:52.448361 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:45:53.448496 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.448463 2580 patch_prober.go:28] interesting pod/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 15:45:53.448642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.448517 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 15:45:53.899234 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.899160 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:53.976331 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976246 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-certs\") pod \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " Sep 10 15:45:53.976331 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976287 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-data\") pod \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " Sep 10 15:45:53.976331 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976306 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-socket\") pod \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " Sep 10 15:45:53.976638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976348 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istiod-ca-cert\") pod \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " Sep 10 15:45:53.976638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976384 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhw5p\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-kube-api-access-zhw5p\") pod \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " Sep 10 15:45:53.976638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976433 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-podinfo\") pod \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " Sep 10 15:45:53.976638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976461 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-credential-socket\") pod \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " Sep 10 15:45:53.976638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976549 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-envoy\") pod \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " Sep 10 15:45:53.976638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976594 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-token\") pod \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\" (UID: \"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff\") " Sep 10 15:45:53.976638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976591 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" (UID: "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:45:53.976638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976603 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" (UID: "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:45:53.977027 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976797 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" (UID: "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:45:53.977027 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.976814 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" (UID: "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:45:53.977132 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.977051 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:53.977132 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.977073 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:53.977132 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.977088 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:53.977132 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.977100 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:53.977132 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.977104 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" (UID: "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:45:53.978521 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.978499 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" (UID: "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:45:53.979169 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.979147 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-kube-api-access-zhw5p" (OuterVolumeSpecName: "kube-api-access-zhw5p") pod "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" (UID: "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff"). InnerVolumeSpecName "kube-api-access-zhw5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:45:53.979210 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.979148 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" (UID: "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:45:53.979210 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:53.979178 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" (UID: "d9668fbe-fd8d-4d60-afde-5a7459e0f9ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:45:54.041903 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.041864 2580 generic.go:356] "Generic (PLEG): container finished" podID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerID="d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9" exitCode=0 Sep 10 15:45:54.042060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.041925 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" Sep 10 15:45:54.042060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.041952 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" event={"ID":"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff","Type":"ContainerDied","Data":"d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9"} Sep 10 15:45:54.042060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.042017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx" event={"ID":"d9668fbe-fd8d-4d60-afde-5a7459e0f9ff","Type":"ContainerDied","Data":"5e8b1753a5e655c49c605b8917b16c5267f95ff36b8e031d8fb3143dadff0a64"} Sep 10 15:45:54.042060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.042048 2580 scope.go:117] "RemoveContainer" containerID="d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9" Sep 10 15:45:54.052376 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.052356 2580 scope.go:117] "RemoveContainer" containerID="0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b" Sep 10 15:45:54.060283 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.060267 2580 scope.go:117] "RemoveContainer" containerID="d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9" Sep 10 15:45:54.060699 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:45:54.060680 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9\": container with ID starting with d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9 not found: ID does not exist" containerID="d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9" Sep 10 15:45:54.060753 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.060707 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9"} err="failed to get container status \"d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9\": rpc error: code = NotFound desc = could not find container \"d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9\": container with ID starting with d98a2a90c06c3a0b22cd372f059c680b21c1b83f29a57df684270243eacfedd9 not found: ID does not exist" Sep 10 15:45:54.060753 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.060724 2580 scope.go:117] "RemoveContainer" containerID="0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b" Sep 10 15:45:54.060958 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:45:54.060940 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b\": container with ID starting with 0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b not found: ID does not exist" containerID="0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b" Sep 10 15:45:54.060998 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.060965 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b"} err="failed to get container status \"0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b\": rpc error: code = NotFound desc = could not find container \"0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b\": container with ID starting with 0c4a191c2c3c8b803fd8e13aa329709c1b5a208928fa35b4c58ad0c28a3c0d4b not found: ID does not exist" Sep 10 15:45:54.065603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.065584 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx"] Sep 10 15:45:54.069189 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.069168 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7083b0df-openshift-default-679cb548c6-zxhmx"] Sep 10 15:45:54.078546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.078527 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:54.078624 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.078548 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhw5p\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-kube-api-access-zhw5p\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:54.078624 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.078559 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:54.078624 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.078568 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:54.078624 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.078576 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:45:54.579089 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.579053 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:54.579900 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.579872 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:54.580996 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:54.580974 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:45:56.029093 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.029059 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" path="/var/lib/kubelet/pods/d9668fbe-fd8d-4d60-afde-5a7459e0f9ff/volumes" Sep 10 15:45:56.261551 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.261510 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx"] Sep 10 15:45:56.261914 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.261901 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" Sep 10 15:45:56.261958 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.261915 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" Sep 10 15:45:56.261958 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.261924 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abcdc54-cb44-4af0-8c87-da47c9358c2d" containerName="payload-pre-processing" Sep 10 15:45:56.261958 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.261930 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abcdc54-cb44-4af0-8c87-da47c9358c2d" containerName="payload-pre-processing" Sep 10 15:45:56.261958 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.261947 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" Sep 10 15:45:56.261958 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.261954 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" Sep 10 15:45:56.262146 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.261964 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="035d3495-525f-4d05-8128-f2b1967de117" containerName="revoke-keys" Sep 10 15:45:56.262146 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.261969 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="035d3495-525f-4d05-8128-f2b1967de117" containerName="revoke-keys" Sep 10 15:45:56.262146 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.262026 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" Sep 10 15:45:56.262146 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.262034 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9668fbe-fd8d-4d60-afde-5a7459e0f9ff" containerName="istio-proxy" Sep 10 15:45:56.262146 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.262041 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abcdc54-cb44-4af0-8c87-da47c9358c2d" containerName="payload-pre-processing" Sep 10 15:45:56.262146 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.262048 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="035d3495-525f-4d05-8128-f2b1967de117" containerName="revoke-keys" Sep 10 15:45:56.265864 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.265845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:45:56.268240 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.268218 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d8392afe\"" Sep 10 15:45:56.268353 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.268270 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d8392afe-dockercfg-ppkvh\"" Sep 10 15:45:56.276311 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.276287 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx"] Sep 10 15:45:56.404997 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.404951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44230c1-0a71-488f-a0de-58f122aa14cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx\" (UID: \"c44230c1-0a71-488f-a0de-58f122aa14cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:45:56.405166 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.405028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78vdq\" (UniqueName: \"kubernetes.io/projected/c44230c1-0a71-488f-a0de-58f122aa14cd-kube-api-access-78vdq\") pod \"payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx\" (UID: \"c44230c1-0a71-488f-a0de-58f122aa14cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:45:56.506477 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.506442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44230c1-0a71-488f-a0de-58f122aa14cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx\" (UID: \"c44230c1-0a71-488f-a0de-58f122aa14cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:45:56.506642 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.506574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78vdq\" (UniqueName: \"kubernetes.io/projected/c44230c1-0a71-488f-a0de-58f122aa14cd-kube-api-access-78vdq\") pod \"payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx\" (UID: \"c44230c1-0a71-488f-a0de-58f122aa14cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:45:56.507112 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.507092 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44230c1-0a71-488f-a0de-58f122aa14cd-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx\" (UID: \"c44230c1-0a71-488f-a0de-58f122aa14cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:45:56.513908 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.513884 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78vdq\" (UniqueName: \"kubernetes.io/projected/c44230c1-0a71-488f-a0de-58f122aa14cd-kube-api-access-78vdq\") pod \"payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx\" (UID: \"c44230c1-0a71-488f-a0de-58f122aa14cd\") " pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:45:56.578008 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.577983 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:45:56.859448 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:56.859408 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx"] Sep 10 15:45:56.861169 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:45:56.861144 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44230c1_0a71_488f_a0de_58f122aa14cd.slice/crio-b5a14bc7172df26fa4ed91df9b699034a544b45f9af9b06c8b9712f076a57c2c WatchSource:0}: Error finding container b5a14bc7172df26fa4ed91df9b699034a544b45f9af9b06c8b9712f076a57c2c: Status 404 returned error can't find the container with id b5a14bc7172df26fa4ed91df9b699034a544b45f9af9b06c8b9712f076a57c2c Sep 10 15:45:57.061809 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:57.061769 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" event={"ID":"c44230c1-0a71-488f-a0de-58f122aa14cd","Type":"ContainerStarted","Data":"d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62"} Sep 10 15:45:57.062173 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:57.061843 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:45:57.062173 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:57.061869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" event={"ID":"c44230c1-0a71-488f-a0de-58f122aa14cd","Type":"ContainerStarted","Data":"b5a14bc7172df26fa4ed91df9b699034a544b45f9af9b06c8b9712f076a57c2c"} Sep 10 15:45:57.077212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:45:57.077165 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" podStartSLOduration=1.077147722 podStartE2EDuration="1.077147722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:57.077012201 +0000 UTC m=+815.639321392" watchObservedRunningTime="2026-09-10 15:45:57.077147722 +0000 UTC m=+815.639456913" Sep 10 15:46:08.069589 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:08.069561 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:46:20.191900 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.191869 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv"] Sep 10 15:46:20.194798 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.194776 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" Sep 10 15:46:20.197055 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.197032 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:46:20.197162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.197052 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:46:20.202591 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.202569 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv"] Sep 10 15:46:20.240483 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.240445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a774f-9987-4e11-b6d5-bf224d161d4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv\" (UID: \"008a774f-9987-4e11-b6d5-bf224d161d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" Sep 10 15:46:20.341725 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.341689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a774f-9987-4e11-b6d5-bf224d161d4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv\" (UID: \"008a774f-9987-4e11-b6d5-bf224d161d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" Sep 10 15:46:20.342341 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.342322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a774f-9987-4e11-b6d5-bf224d161d4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv\" (UID: \"008a774f-9987-4e11-b6d5-bf224d161d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" Sep 10 15:46:20.507737 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.507652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" Sep 10 15:46:20.779351 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:20.779322 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv"] Sep 10 15:46:20.780711 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:46:20.780677 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008a774f_9987_4e11_b6d5_bf224d161d4e.slice/crio-5d9f577b5f2bab14e99ab2def3b0d902b5d2272f83540ca1d09b4bac61d94e67 WatchSource:0}: Error finding container 5d9f577b5f2bab14e99ab2def3b0d902b5d2272f83540ca1d09b4bac61d94e67: Status 404 returned error can't find the container with id 5d9f577b5f2bab14e99ab2def3b0d902b5d2272f83540ca1d09b4bac61d94e67 Sep 10 15:46:21.192917 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:21.192886 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" event={"ID":"008a774f-9987-4e11-b6d5-bf224d161d4e","Type":"ContainerStarted","Data":"17844b0d57ef731b5ccfec4c34ff479045ccdc6d7de8428d216f70ce741d4f6d"} Sep 10 15:46:21.193281 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:21.192938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" event={"ID":"008a774f-9987-4e11-b6d5-bf224d161d4e","Type":"ContainerStarted","Data":"5d9f577b5f2bab14e99ab2def3b0d902b5d2272f83540ca1d09b4bac61d94e67"} Sep 10 15:46:21.208401 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:21.208341 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" podStartSLOduration=1.208322706 podStartE2EDuration="1.208322706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:21.208305157 +0000 UTC m=+839.770614348" watchObservedRunningTime="2026-09-10 15:46:21.208322706 +0000 UTC m=+839.770631899" Sep 10 15:46:22.199974 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:22.199943 2580 generic.go:356] "Generic (PLEG): container finished" podID="008a774f-9987-4e11-b6d5-bf224d161d4e" containerID="17844b0d57ef731b5ccfec4c34ff479045ccdc6d7de8428d216f70ce741d4f6d" exitCode=0 Sep 10 15:46:22.200380 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:22.200019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" event={"ID":"008a774f-9987-4e11-b6d5-bf224d161d4e","Type":"ContainerDied","Data":"17844b0d57ef731b5ccfec4c34ff479045ccdc6d7de8428d216f70ce741d4f6d"} Sep 10 15:46:23.334935 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:23.334911 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" Sep 10 15:46:23.371106 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:23.371070 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a774f-9987-4e11-b6d5-bf224d161d4e-maas-api-service-ca\") pod \"008a774f-9987-4e11-b6d5-bf224d161d4e\" (UID: \"008a774f-9987-4e11-b6d5-bf224d161d4e\") " Sep 10 15:46:23.371528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:23.371499 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008a774f-9987-4e11-b6d5-bf224d161d4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "008a774f-9987-4e11-b6d5-bf224d161d4e" (UID: "008a774f-9987-4e11-b6d5-bf224d161d4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:23.472181 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:23.472095 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/008a774f-9987-4e11-b6d5-bf224d161d4e-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:46:24.219253 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:24.219220 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" event={"ID":"008a774f-9987-4e11-b6d5-bf224d161d4e","Type":"ContainerDied","Data":"5d9f577b5f2bab14e99ab2def3b0d902b5d2272f83540ca1d09b4bac61d94e67"} Sep 10 15:46:24.219253 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:24.219241 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv" Sep 10 15:46:24.219470 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:24.219274 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d9f577b5f2bab14e99ab2def3b0d902b5d2272f83540ca1d09b4bac61d94e67" Sep 10 15:46:26.897986 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:26.897949 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx"] Sep 10 15:46:26.898511 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:26.898260 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" podUID="c44230c1-0a71-488f-a0de-58f122aa14cd" containerName="payload-processing" containerID="cri-o://d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62" gracePeriod=30 Sep 10 15:46:27.063210 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.063170 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv"] Sep 10 15:46:27.067009 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.066983 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d8392afe-pwkvv"] Sep 10 15:46:27.154965 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.154894 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:46:27.210992 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.210963 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78vdq\" (UniqueName: \"kubernetes.io/projected/c44230c1-0a71-488f-a0de-58f122aa14cd-kube-api-access-78vdq\") pod \"c44230c1-0a71-488f-a0de-58f122aa14cd\" (UID: \"c44230c1-0a71-488f-a0de-58f122aa14cd\") " Sep 10 15:46:27.211180 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.211000 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44230c1-0a71-488f-a0de-58f122aa14cd-plugins-config-volume\") pod \"c44230c1-0a71-488f-a0de-58f122aa14cd\" (UID: \"c44230c1-0a71-488f-a0de-58f122aa14cd\") " Sep 10 15:46:27.211468 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.211440 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44230c1-0a71-488f-a0de-58f122aa14cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c44230c1-0a71-488f-a0de-58f122aa14cd" (UID: "c44230c1-0a71-488f-a0de-58f122aa14cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:27.214628 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.214604 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44230c1-0a71-488f-a0de-58f122aa14cd-kube-api-access-78vdq" (OuterVolumeSpecName: "kube-api-access-78vdq") pod "c44230c1-0a71-488f-a0de-58f122aa14cd" (UID: "c44230c1-0a71-488f-a0de-58f122aa14cd"). InnerVolumeSpecName "kube-api-access-78vdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:27.241111 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.241080 2580 generic.go:356] "Generic (PLEG): container finished" podID="c44230c1-0a71-488f-a0de-58f122aa14cd" containerID="d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62" exitCode=0 Sep 10 15:46:27.241278 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.241127 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" Sep 10 15:46:27.241278 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.241137 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" event={"ID":"c44230c1-0a71-488f-a0de-58f122aa14cd","Type":"ContainerDied","Data":"d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62"} Sep 10 15:46:27.241278 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.241191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx" event={"ID":"c44230c1-0a71-488f-a0de-58f122aa14cd","Type":"ContainerDied","Data":"b5a14bc7172df26fa4ed91df9b699034a544b45f9af9b06c8b9712f076a57c2c"} Sep 10 15:46:27.241278 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.241222 2580 scope.go:117] "RemoveContainer" containerID="d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62" Sep 10 15:46:27.250721 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.250704 2580 scope.go:117] "RemoveContainer" containerID="d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62" Sep 10 15:46:27.250999 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:46:27.250978 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62\": container with ID starting with d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62 not found: ID does not exist" containerID="d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62" Sep 10 15:46:27.251044 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.251008 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62"} err="failed to get container status \"d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62\": rpc error: code = NotFound desc = could not find container \"d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62\": container with ID starting with d0f90f472808028b2eb6cb0896c9ddeb39d8f86e7108802f6b86badbdd210c62 not found: ID does not exist" Sep 10 15:46:27.269147 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.269118 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx"] Sep 10 15:46:27.275825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.275796 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d8392afe-6cdffc7766-j66lx"] Sep 10 15:46:27.312974 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.312941 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78vdq\" (UniqueName: \"kubernetes.io/projected/c44230c1-0a71-488f-a0de-58f122aa14cd-kube-api-access-78vdq\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:46:27.312974 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.312972 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44230c1-0a71-488f-a0de-58f122aa14cd-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:46:27.937659 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.937626 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79"] Sep 10 15:46:27.938260 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.938148 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44230c1-0a71-488f-a0de-58f122aa14cd" containerName="payload-processing" Sep 10 15:46:27.938260 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.938172 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44230c1-0a71-488f-a0de-58f122aa14cd" containerName="payload-processing" Sep 10 15:46:27.938260 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.938183 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="008a774f-9987-4e11-b6d5-bf224d161d4e" containerName="revoke-keys" Sep 10 15:46:27.938260 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.938192 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="008a774f-9987-4e11-b6d5-bf224d161d4e" containerName="revoke-keys" Sep 10 15:46:27.938569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.938301 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44230c1-0a71-488f-a0de-58f122aa14cd" containerName="payload-processing" Sep 10 15:46:27.938569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.938312 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="008a774f-9987-4e11-b6d5-bf224d161d4e" containerName="revoke-keys" Sep 10 15:46:27.941483 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.941464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:27.944019 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.943997 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-96d7ee49-openshift-default-dockercfg-5vj4h\"" Sep 10 15:46:27.953327 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:27.953304 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79"] Sep 10 15:46:28.019408 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.019371 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfd4b0e-0334-4c62-be04-579a38364573-istio-podinfo\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.019408 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.019411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-envoy\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.019712 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.019469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-socket\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.019712 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.019527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z885c\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-kube-api-access-z885c\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.019712 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.019547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfd4b0e-0334-4c62-be04-579a38364573-istiod-ca-cert\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.019712 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.019566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-credential-socket\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.019860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.019705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-data\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.019860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.019776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-istio-token\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.019860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.019834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-certs\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.031051 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.031003 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="008a774f-9987-4e11-b6d5-bf224d161d4e" path="/var/lib/kubelet/pods/008a774f-9987-4e11-b6d5-bf224d161d4e/volumes" Sep 10 15:46:28.032203 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.031547 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44230c1-0a71-488f-a0de-58f122aa14cd" path="/var/lib/kubelet/pods/c44230c1-0a71-488f-a0de-58f122aa14cd/volumes" Sep 10 15:46:28.120873 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.120838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfd4b0e-0334-4c62-be04-579a38364573-istio-podinfo\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.120873 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.120874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-envoy\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.120900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-socket\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.120934 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z885c\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-kube-api-access-z885c\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.120965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfd4b0e-0334-4c62-be04-579a38364573-istiod-ca-cert\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.120992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-credential-socket\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.121074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-data\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121360 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.121124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-istio-token\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121360 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.121186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-certs\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121729 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.121705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-socket\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121857 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.121747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-data\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121857 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.121775 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-certs\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121857 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.121798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfd4b0e-0334-4c62-be04-579a38364573-istiod-ca-cert\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.121973 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.121949 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-credential-socket\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.123298 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.123283 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-envoy\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.123638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.123619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfd4b0e-0334-4c62-be04-579a38364573-istio-podinfo\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.131149 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.131126 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-istio-token\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.131405 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.131387 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z885c\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-kube-api-access-z885c\") pod \"e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.254022 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.253937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:28.529251 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.529179 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79"] Sep 10 15:46:28.532824 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:46:28.532798 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfd4b0e_0334_4c62_be04_579a38364573.slice/crio-83cc8aa468fe12558bb762d390a19fef1785821f40fcded31b7c7a40eda1f8c1 WatchSource:0}: Error finding container 83cc8aa468fe12558bb762d390a19fef1785821f40fcded31b7c7a40eda1f8c1: Status 404 returned error can't find the container with id 83cc8aa468fe12558bb762d390a19fef1785821f40fcded31b7c7a40eda1f8c1 Sep 10 15:46:28.535488 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.535459 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:46:28.535563 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.535536 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:46:28.535600 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:28.535561 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:46:29.255896 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:29.255864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" event={"ID":"5bfd4b0e-0334-4c62-be04-579a38364573","Type":"ContainerStarted","Data":"f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91"} Sep 10 15:46:29.256280 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:29.255911 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" event={"ID":"5bfd4b0e-0334-4c62-be04-579a38364573","Type":"ContainerStarted","Data":"83cc8aa468fe12558bb762d390a19fef1785821f40fcded31b7c7a40eda1f8c1"} Sep 10 15:46:29.277263 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:29.277217 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" podStartSLOduration=2.277201017 podStartE2EDuration="2.277201017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:29.275315638 +0000 UTC m=+847.837624839" watchObservedRunningTime="2026-09-10 15:46:29.277201017 +0000 UTC m=+847.839510208" Sep 10 15:46:30.254975 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:30.254946 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:30.260616 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:30.260586 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:30.262692 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:30.262664 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:30.263538 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:30.263517 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:32.381821 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.381786 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl"] Sep 10 15:46:32.385017 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.384994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:32.387164 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.387144 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-96d7ee49-dockercfg-hz8v4\"" Sep 10 15:46:32.387244 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.387167 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-96d7ee49\"" Sep 10 15:46:32.394589 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.394567 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl"] Sep 10 15:46:32.467908 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.466505 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44wd\" (UniqueName: \"kubernetes.io/projected/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-kube-api-access-q44wd\") pod \"payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl\" (UID: \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\") " pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:32.467908 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.466675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-plugins-config-volume\") pod \"payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl\" (UID: \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\") " pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:32.567569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.567527 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-plugins-config-volume\") pod \"payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl\" (UID: \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\") " pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:32.567759 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.567714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44wd\" (UniqueName: \"kubernetes.io/projected/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-kube-api-access-q44wd\") pod \"payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl\" (UID: \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\") " pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:32.568279 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.568256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-plugins-config-volume\") pod \"payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl\" (UID: \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\") " pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:32.575598 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.575574 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44wd\" (UniqueName: \"kubernetes.io/projected/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-kube-api-access-q44wd\") pod \"payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl\" (UID: \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\") " pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:32.698112 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.698027 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:32.972083 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:32.972055 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl"] Sep 10 15:46:32.974108 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:46:32.974067 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b31ccf_6a2a_4fcd_b69a_69a8227d4dcb.slice/crio-d57b4c8101a05256b51cbc8fad923efb342c8810047535c74bba58c01778d451 WatchSource:0}: Error finding container d57b4c8101a05256b51cbc8fad923efb342c8810047535c74bba58c01778d451: Status 404 returned error can't find the container with id d57b4c8101a05256b51cbc8fad923efb342c8810047535c74bba58c01778d451 Sep 10 15:46:33.281082 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:33.280996 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" event={"ID":"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb","Type":"ContainerStarted","Data":"44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec"} Sep 10 15:46:33.281082 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:33.281049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" event={"ID":"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb","Type":"ContainerStarted","Data":"d57b4c8101a05256b51cbc8fad923efb342c8810047535c74bba58c01778d451"} Sep 10 15:46:33.281280 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:33.281108 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:33.297625 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:33.297576 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" podStartSLOduration=1.29756092 podStartE2EDuration="1.29756092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:33.29561247 +0000 UTC m=+851.857921672" watchObservedRunningTime="2026-09-10 15:46:33.29756092 +0000 UTC m=+851.859870109" Sep 10 15:46:44.288990 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:44.288959 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:46:56.339049 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.339017 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc"] Sep 10 15:46:56.341801 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.341777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" Sep 10 15:46:56.344051 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.344024 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:46:56.344167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.344027 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:46:56.349077 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.349055 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc"] Sep 10 15:46:56.410751 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.410721 2580 generic.go:356] "Generic (PLEG): container finished" podID="5bfd4b0e-0334-4c62-be04-579a38364573" containerID="f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91" exitCode=0 Sep 10 15:46:56.410912 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.410789 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" event={"ID":"5bfd4b0e-0334-4c62-be04-579a38364573","Type":"ContainerDied","Data":"f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91"} Sep 10 15:46:56.411227 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.411212 2580 scope.go:117] "RemoveContainer" containerID="f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91" Sep 10 15:46:56.412355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.412329 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:46:56.412484 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.412409 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:46:56.412484 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.412474 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:46:56.417285 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.417254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904ec0e1-d07b-45a2-839f-6a4428b00d04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc\" (UID: \"904ec0e1-d07b-45a2-839f-6a4428b00d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" Sep 10 15:46:56.519214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.519032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904ec0e1-d07b-45a2-839f-6a4428b00d04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc\" (UID: \"904ec0e1-d07b-45a2-839f-6a4428b00d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" Sep 10 15:46:56.519818 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.519790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904ec0e1-d07b-45a2-839f-6a4428b00d04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc\" (UID: \"904ec0e1-d07b-45a2-839f-6a4428b00d04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" Sep 10 15:46:56.654149 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.654117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" Sep 10 15:46:56.932466 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:56.932373 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc"] Sep 10 15:46:56.934778 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:46:56.934747 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904ec0e1_d07b_45a2_839f_6a4428b00d04.slice/crio-dd6e0c9393791c2bf25cfe6a99084ca3cf8a8cdb4e268868a7d29c8686823896 WatchSource:0}: Error finding container dd6e0c9393791c2bf25cfe6a99084ca3cf8a8cdb4e268868a7d29c8686823896: Status 404 returned error can't find the container with id dd6e0c9393791c2bf25cfe6a99084ca3cf8a8cdb4e268868a7d29c8686823896 Sep 10 15:46:57.423624 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:57.423579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" event={"ID":"5bfd4b0e-0334-4c62-be04-579a38364573","Type":"ContainerStarted","Data":"eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01"} Sep 10 15:46:57.426775 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:57.426739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" event={"ID":"904ec0e1-d07b-45a2-839f-6a4428b00d04","Type":"ContainerStarted","Data":"fb9b5ab295f8e80e8aa4530200e0b1aa981d4cc5247ae876c4b37abe9f11a216"} Sep 10 15:46:57.426937 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:57.426799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" event={"ID":"904ec0e1-d07b-45a2-839f-6a4428b00d04","Type":"ContainerStarted","Data":"dd6e0c9393791c2bf25cfe6a99084ca3cf8a8cdb4e268868a7d29c8686823896"} Sep 10 15:46:57.458124 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:57.458065 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" podStartSLOduration=1.458049302 podStartE2EDuration="1.458049302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:57.456722996 +0000 UTC m=+876.019032187" watchObservedRunningTime="2026-09-10 15:46:57.458049302 +0000 UTC m=+876.020358491" Sep 10 15:46:58.254386 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:58.254354 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:46:58.255684 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:58.255654 2580 patch_prober.go:28] interesting pod/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 10 15:46:58.255814 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:58.255707 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 10 15:46:58.438957 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:58.438923 2580 generic.go:356] "Generic (PLEG): container finished" podID="904ec0e1-d07b-45a2-839f-6a4428b00d04" containerID="fb9b5ab295f8e80e8aa4530200e0b1aa981d4cc5247ae876c4b37abe9f11a216" exitCode=0 Sep 10 15:46:58.439475 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:58.439022 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" event={"ID":"904ec0e1-d07b-45a2-839f-6a4428b00d04","Type":"ContainerDied","Data":"fb9b5ab295f8e80e8aa4530200e0b1aa981d4cc5247ae876c4b37abe9f11a216"} Sep 10 15:46:59.254363 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:59.254321 2580 patch_prober.go:28] interesting pod/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 10 15:46:59.254666 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:59.254395 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 10 15:46:59.600143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:59.600119 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" Sep 10 15:46:59.657639 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:59.657132 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904ec0e1-d07b-45a2-839f-6a4428b00d04-maas-api-service-ca\") pod \"904ec0e1-d07b-45a2-839f-6a4428b00d04\" (UID: \"904ec0e1-d07b-45a2-839f-6a4428b00d04\") " Sep 10 15:46:59.657639 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:59.657502 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904ec0e1-d07b-45a2-839f-6a4428b00d04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "904ec0e1-d07b-45a2-839f-6a4428b00d04" (UID: "904ec0e1-d07b-45a2-839f-6a4428b00d04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:46:59.657639 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:46:59.657606 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/904ec0e1-d07b-45a2-839f-6a4428b00d04-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:00.255512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:00.255377 2580 patch_prober.go:28] interesting pod/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 10 15:47:00.255512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:00.255459 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 10 15:47:00.263447 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:00.263405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:47:00.457004 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:00.456974 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" Sep 10 15:47:00.457326 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:00.456993 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc" event={"ID":"904ec0e1-d07b-45a2-839f-6a4428b00d04","Type":"ContainerDied","Data":"dd6e0c9393791c2bf25cfe6a99084ca3cf8a8cdb4e268868a7d29c8686823896"} Sep 10 15:47:00.457498 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:00.457334 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd6e0c9393791c2bf25cfe6a99084ca3cf8a8cdb4e268868a7d29c8686823896" Sep 10 15:47:01.255314 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:01.255269 2580 patch_prober.go:28] interesting pod/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 10 15:47:01.255734 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:01.255366 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 10 15:47:02.254629 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.254590 2580 patch_prober.go:28] interesting pod/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Sep 10 15:47:02.254813 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.254647 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Sep 10 15:47:02.674105 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.674070 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl"] Sep 10 15:47:02.674499 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.674317 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" podUID="17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb" containerName="payload-processing" containerID="cri-o://44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec" gracePeriod=30 Sep 10 15:47:02.854536 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.854500 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc"] Sep 10 15:47:02.857591 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.857551 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-96d7ee49-qmnsc"] Sep 10 15:47:02.962167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.961792 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:47:02.993149 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.993117 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q44wd\" (UniqueName: \"kubernetes.io/projected/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-kube-api-access-q44wd\") pod \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\" (UID: \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\") " Sep 10 15:47:02.993319 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.993174 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-plugins-config-volume\") pod \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\" (UID: \"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb\") " Sep 10 15:47:02.993579 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.993556 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb" (UID: "17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:47:02.995855 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:02.995829 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-kube-api-access-q44wd" (OuterVolumeSpecName: "kube-api-access-q44wd") pod "17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb" (UID: "17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb"). InnerVolumeSpecName "kube-api-access-q44wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:47:03.094698 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.094647 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44wd\" (UniqueName: \"kubernetes.io/projected/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-kube-api-access-q44wd\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:03.094698 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.094682 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:03.258363 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.258274 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:47:03.259330 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.259308 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:47:03.474042 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.474011 2580 generic.go:356] "Generic (PLEG): container finished" podID="17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb" containerID="44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec" exitCode=0 Sep 10 15:47:03.474205 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.474060 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" Sep 10 15:47:03.474205 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.474097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" event={"ID":"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb","Type":"ContainerDied","Data":"44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec"} Sep 10 15:47:03.474205 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.474152 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl" event={"ID":"17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb","Type":"ContainerDied","Data":"d57b4c8101a05256b51cbc8fad923efb342c8810047535c74bba58c01778d451"} Sep 10 15:47:03.474205 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.474181 2580 scope.go:117] "RemoveContainer" containerID="44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec" Sep 10 15:47:03.483338 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.483322 2580 scope.go:117] "RemoveContainer" containerID="44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec" Sep 10 15:47:03.483616 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:47:03.483596 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec\": container with ID starting with 44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec not found: ID does not exist" containerID="44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec" Sep 10 15:47:03.483677 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.483627 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec"} err="failed to get container status \"44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec\": rpc error: code = NotFound desc = could not find container \"44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec\": container with ID starting with 44d87bda722c572ed8c572fbb18cabf566c71aecb5295f8e32b6cbc9a98c22ec not found: ID does not exist" Sep 10 15:47:03.501596 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.501569 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl"] Sep 10 15:47:03.505323 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.505296 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-96d7ee49-67d8676db5-lhxhl"] Sep 10 15:47:03.961742 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:03.961709 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79"] Sep 10 15:47:04.028893 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.028862 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb" path="/var/lib/kubelet/pods/17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb/volumes" Sep 10 15:47:04.029203 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.029189 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="904ec0e1-d07b-45a2-839f-6a4428b00d04" path="/var/lib/kubelet/pods/904ec0e1-d07b-45a2-839f-6a4428b00d04/volumes" Sep 10 15:47:04.481170 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.481105 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" containerID="cri-o://eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01" gracePeriod=30 Sep 10 15:47:04.677762 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.677731 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6"] Sep 10 15:47:04.678130 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.678119 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb" containerName="payload-processing" Sep 10 15:47:04.678184 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.678132 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb" containerName="payload-processing" Sep 10 15:47:04.678184 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.678151 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="904ec0e1-d07b-45a2-839f-6a4428b00d04" containerName="revoke-keys" Sep 10 15:47:04.678184 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.678158 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="904ec0e1-d07b-45a2-839f-6a4428b00d04" containerName="revoke-keys" Sep 10 15:47:04.678294 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.678216 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b31ccf-6a2a-4fcd-b69a-69a8227d4dcb" containerName="payload-processing" Sep 10 15:47:04.678294 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.678226 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="904ec0e1-d07b-45a2-839f-6a4428b00d04" containerName="revoke-keys" Sep 10 15:47:04.681761 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.681731 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.683894 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.683875 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ff2a412a-openshift-default-dockercfg-jg6nh\"" Sep 10 15:47:04.694011 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.693970 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6"] Sep 10 15:47:04.811280 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.811184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-data\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.811280 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.811243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv5tq\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-kube-api-access-xv5tq\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.811510 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.811360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-credential-socket\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.811510 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.811409 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34a398-bd73-4a82-b402-a2883fd36242-istio-podinfo\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.811510 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.811445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34a398-bd73-4a82-b402-a2883fd36242-istiod-ca-cert\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.811510 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.811482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-socket\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.811791 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.811590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-istio-token\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.811791 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.811651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-envoy\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.811791 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.811756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-certs\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.912563 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.912518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-istio-token\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.912563 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.912557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-envoy\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.912817 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.912605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-certs\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.912817 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.912633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-data\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.912817 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.912648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv5tq\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-kube-api-access-xv5tq\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.912817 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.912676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-credential-socket\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.912817 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.912726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34a398-bd73-4a82-b402-a2883fd36242-istio-podinfo\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.912817 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.912762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34a398-bd73-4a82-b402-a2883fd36242-istiod-ca-cert\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.912817 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.912792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-socket\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.913158 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.913047 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-certs\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.913158 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.913099 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-credential-socket\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.913241 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.913153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-data\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.913241 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.913215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-socket\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.913473 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.913415 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34a398-bd73-4a82-b402-a2883fd36242-istiod-ca-cert\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.915169 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.915147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-envoy\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.915169 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.915163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34a398-bd73-4a82-b402-a2883fd36242-istio-podinfo\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.925354 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.925332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-istio-token\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.925649 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.925619 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv5tq\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-kube-api-access-xv5tq\") pod \"e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:04.994489 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:04.994453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:05.277359 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:05.277332 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6"] Sep 10 15:47:05.278755 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:47:05.278727 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a34a398_bd73_4a82_b402_a2883fd36242.slice/crio-33a39d8824f63042855789a4f1224a219ac35b295322f30b999d26939a0573e7 WatchSource:0}: Error finding container 33a39d8824f63042855789a4f1224a219ac35b295322f30b999d26939a0573e7: Status 404 returned error can't find the container with id 33a39d8824f63042855789a4f1224a219ac35b295322f30b999d26939a0573e7 Sep 10 15:47:05.280923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:05.280895 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:47:05.280999 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:05.280958 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:47:05.280999 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:05.280984 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:47:05.488919 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:05.488885 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" event={"ID":"5a34a398-bd73-4a82-b402-a2883fd36242","Type":"ContainerStarted","Data":"a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9"} Sep 10 15:47:05.489125 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:05.488939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" event={"ID":"5a34a398-bd73-4a82-b402-a2883fd36242","Type":"ContainerStarted","Data":"33a39d8824f63042855789a4f1224a219ac35b295322f30b999d26939a0573e7"} Sep 10 15:47:05.510121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:05.510061 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" podStartSLOduration=1.510039097 podStartE2EDuration="1.510039097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:05.507911474 +0000 UTC m=+884.070220666" watchObservedRunningTime="2026-09-10 15:47:05.510039097 +0000 UTC m=+884.072348290" Sep 10 15:47:05.995492 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:05.995454 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:07.000522 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:07.000490 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:07.506019 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:07.505988 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:07.507010 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:07.506987 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:09.758761 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.758703 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:47:09.879746 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.879713 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z885c\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-kube-api-access-z885c\") pod \"5bfd4b0e-0334-4c62-be04-579a38364573\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " Sep 10 15:47:09.879926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.879760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-credential-socket\") pod \"5bfd4b0e-0334-4c62-be04-579a38364573\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " Sep 10 15:47:09.879926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.879824 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-socket\") pod \"5bfd4b0e-0334-4c62-be04-579a38364573\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " Sep 10 15:47:09.879926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.879866 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfd4b0e-0334-4c62-be04-579a38364573-istio-podinfo\") pod \"5bfd4b0e-0334-4c62-be04-579a38364573\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " Sep 10 15:47:09.879926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.879896 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-certs\") pod \"5bfd4b0e-0334-4c62-be04-579a38364573\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " Sep 10 15:47:09.880147 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880019 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-istio-token\") pod \"5bfd4b0e-0334-4c62-be04-579a38364573\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " Sep 10 15:47:09.880147 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880087 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfd4b0e-0334-4c62-be04-579a38364573-istiod-ca-cert\") pod \"5bfd4b0e-0334-4c62-be04-579a38364573\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " Sep 10 15:47:09.880147 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880088 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5bfd4b0e-0334-4c62-be04-579a38364573" (UID: "5bfd4b0e-0334-4c62-be04-579a38364573"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:09.880304 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880165 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5bfd4b0e-0334-4c62-be04-579a38364573" (UID: "5bfd4b0e-0334-4c62-be04-579a38364573"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:09.880304 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880196 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5bfd4b0e-0334-4c62-be04-579a38364573" (UID: "5bfd4b0e-0334-4c62-be04-579a38364573"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:09.880304 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880228 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-envoy\") pod \"5bfd4b0e-0334-4c62-be04-579a38364573\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " Sep 10 15:47:09.880304 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880264 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-data\") pod \"5bfd4b0e-0334-4c62-be04-579a38364573\" (UID: \"5bfd4b0e-0334-4c62-be04-579a38364573\") " Sep 10 15:47:09.880568 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880545 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-data" (OuterVolumeSpecName: "istio-data") pod "5bfd4b0e-0334-4c62-be04-579a38364573" (UID: "5bfd4b0e-0334-4c62-be04-579a38364573"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:09.880629 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880612 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bfd4b0e-0334-4c62-be04-579a38364573-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5bfd4b0e-0334-4c62-be04-579a38364573" (UID: "5bfd4b0e-0334-4c62-be04-579a38364573"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:47:09.880768 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880743 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:09.880768 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880769 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bfd4b0e-0334-4c62-be04-579a38364573-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:09.880950 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880782 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:09.880950 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880795 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:09.880950 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.880808 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:09.882125 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.882094 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-istio-token" (OuterVolumeSpecName: "istio-token") pod "5bfd4b0e-0334-4c62-be04-579a38364573" (UID: "5bfd4b0e-0334-4c62-be04-579a38364573"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:47:09.882211 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.882142 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-kube-api-access-z885c" (OuterVolumeSpecName: "kube-api-access-z885c") pod "5bfd4b0e-0334-4c62-be04-579a38364573" (UID: "5bfd4b0e-0334-4c62-be04-579a38364573"). InnerVolumeSpecName "kube-api-access-z885c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:47:09.882211 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.882147 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bfd4b0e-0334-4c62-be04-579a38364573-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5bfd4b0e-0334-4c62-be04-579a38364573" (UID: "5bfd4b0e-0334-4c62-be04-579a38364573"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:47:09.882211 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.882197 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5bfd4b0e-0334-4c62-be04-579a38364573" (UID: "5bfd4b0e-0334-4c62-be04-579a38364573"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:09.982005 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.981961 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bfd4b0e-0334-4c62-be04-579a38364573-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:09.982005 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.982001 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z885c\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-kube-api-access-z885c\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:09.982005 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.982013 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bfd4b0e-0334-4c62-be04-579a38364573-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:09.982005 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:09.982021 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bfd4b0e-0334-4c62-be04-579a38364573-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:10.526014 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.525987 2580 generic.go:356] "Generic (PLEG): container finished" podID="5bfd4b0e-0334-4c62-be04-579a38364573" containerID="eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01" exitCode=0 Sep 10 15:47:10.526159 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.526032 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" Sep 10 15:47:10.526159 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.526035 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" event={"ID":"5bfd4b0e-0334-4c62-be04-579a38364573","Type":"ContainerDied","Data":"eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01"} Sep 10 15:47:10.526282 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.526173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79" event={"ID":"5bfd4b0e-0334-4c62-be04-579a38364573","Type":"ContainerDied","Data":"83cc8aa468fe12558bb762d390a19fef1785821f40fcded31b7c7a40eda1f8c1"} Sep 10 15:47:10.526282 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.526213 2580 scope.go:117] "RemoveContainer" containerID="eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01" Sep 10 15:47:10.537952 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.537928 2580 scope.go:117] "RemoveContainer" containerID="f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91" Sep 10 15:47:10.544521 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.544495 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79"] Sep 10 15:47:10.547095 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.547073 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-96d7ee49-openshift-default-d4985ff7d-p5h79"] Sep 10 15:47:10.547750 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.547732 2580 scope.go:117] "RemoveContainer" containerID="eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01" Sep 10 15:47:10.548017 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:47:10.547995 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01\": container with ID starting with eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01 not found: ID does not exist" containerID="eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01" Sep 10 15:47:10.548121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.548023 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01"} err="failed to get container status \"eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01\": rpc error: code = NotFound desc = could not find container \"eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01\": container with ID starting with eb35426773f7dea49258a2e05a3d9132e9488c5a10d9e336ae3f96efc61cbe01 not found: ID does not exist" Sep 10 15:47:10.548121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.548045 2580 scope.go:117] "RemoveContainer" containerID="f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91" Sep 10 15:47:10.548297 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:47:10.548271 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91\": container with ID starting with f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91 not found: ID does not exist" containerID="f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91" Sep 10 15:47:10.548345 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:10.548303 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91"} err="failed to get container status \"f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91\": rpc error: code = NotFound desc = could not find container \"f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91\": container with ID starting with f490d8101d3e05e7d255e84f6219ea55f28c87510625016b1828ef13a7a15f91 not found: ID does not exist" Sep 10 15:47:12.030692 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:12.030663 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" path="/var/lib/kubelet/pods/5bfd4b0e-0334-4c62-be04-579a38364573/volumes" Sep 10 15:47:22.000245 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:22.000169 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:47:22.001689 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:22.001669 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:47:23.075479 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.075439 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2"] Sep 10 15:47:23.075954 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.075939 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" Sep 10 15:47:23.075998 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.075963 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" Sep 10 15:47:23.076054 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.076045 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" Sep 10 15:47:23.076100 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.076058 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" Sep 10 15:47:23.079305 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.079287 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" Sep 10 15:47:23.081873 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.081767 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:47:23.082610 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.082460 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:47:23.085149 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.085124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2"] Sep 10 15:47:23.127446 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.126248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76856e53-f04a-4f25-aea5-b6a4b9464a1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-d5048f-sp6m2\" (UID: \"76856e53-f04a-4f25-aea5-b6a4b9464a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" Sep 10 15:47:23.227081 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.227034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76856e53-f04a-4f25-aea5-b6a4b9464a1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-d5048f-sp6m2\" (UID: \"76856e53-f04a-4f25-aea5-b6a4b9464a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" Sep 10 15:47:23.228045 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.228013 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76856e53-f04a-4f25-aea5-b6a4b9464a1d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-d5048f-sp6m2\" (UID: \"76856e53-f04a-4f25-aea5-b6a4b9464a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" Sep 10 15:47:23.391860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.391829 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" Sep 10 15:47:23.682210 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:23.682183 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2"] Sep 10 15:47:23.684381 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:47:23.684355 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76856e53_f04a_4f25_aea5_b6a4b9464a1d.slice/crio-351adb8b2446caf1e4c0f9b8df61ab4de61bdfa11f6a4d1e633f7714c8f40b67 WatchSource:0}: Error finding container 351adb8b2446caf1e4c0f9b8df61ab4de61bdfa11f6a4d1e633f7714c8f40b67: Status 404 returned error can't find the container with id 351adb8b2446caf1e4c0f9b8df61ab4de61bdfa11f6a4d1e633f7714c8f40b67 Sep 10 15:47:24.601222 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:24.601183 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" event={"ID":"76856e53-f04a-4f25-aea5-b6a4b9464a1d","Type":"ContainerStarted","Data":"781dbfda1daaa5ac9913df0fff5f314daea62ea8ecd77fd1b6846db8535ca282"} Sep 10 15:47:24.601635 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:24.601234 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" event={"ID":"76856e53-f04a-4f25-aea5-b6a4b9464a1d","Type":"ContainerStarted","Data":"351adb8b2446caf1e4c0f9b8df61ab4de61bdfa11f6a4d1e633f7714c8f40b67"} Sep 10 15:47:24.625883 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:24.625815 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" podStartSLOduration=1.625792878 podStartE2EDuration="1.625792878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:24.621378927 +0000 UTC m=+903.183688118" watchObservedRunningTime="2026-09-10 15:47:24.625792878 +0000 UTC m=+903.188102071" Sep 10 15:47:25.608877 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:25.608838 2580 generic.go:356] "Generic (PLEG): container finished" podID="76856e53-f04a-4f25-aea5-b6a4b9464a1d" containerID="781dbfda1daaa5ac9913df0fff5f314daea62ea8ecd77fd1b6846db8535ca282" exitCode=0 Sep 10 15:47:25.609288 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:25.608892 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" event={"ID":"76856e53-f04a-4f25-aea5-b6a4b9464a1d","Type":"ContainerDied","Data":"781dbfda1daaa5ac9913df0fff5f314daea62ea8ecd77fd1b6846db8535ca282"} Sep 10 15:47:26.752006 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:26.751983 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" Sep 10 15:47:26.766320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:26.766297 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76856e53-f04a-4f25-aea5-b6a4b9464a1d-maas-api-service-ca\") pod \"76856e53-f04a-4f25-aea5-b6a4b9464a1d\" (UID: \"76856e53-f04a-4f25-aea5-b6a4b9464a1d\") " Sep 10 15:47:26.766985 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:26.766953 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76856e53-f04a-4f25-aea5-b6a4b9464a1d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76856e53-f04a-4f25-aea5-b6a4b9464a1d" (UID: "76856e53-f04a-4f25-aea5-b6a4b9464a1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:47:26.868375 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:26.868290 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76856e53-f04a-4f25-aea5-b6a4b9464a1d-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:27.624559 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:27.624519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" event={"ID":"76856e53-f04a-4f25-aea5-b6a4b9464a1d","Type":"ContainerDied","Data":"351adb8b2446caf1e4c0f9b8df61ab4de61bdfa11f6a4d1e633f7714c8f40b67"} Sep 10 15:47:27.624559 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:27.624546 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2" Sep 10 15:47:27.624878 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:27.624597 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="351adb8b2446caf1e4c0f9b8df61ab4de61bdfa11f6a4d1e633f7714c8f40b67" Sep 10 15:47:28.630958 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:28.630923 2580 generic.go:356] "Generic (PLEG): container finished" podID="5a34a398-bd73-4a82-b402-a2883fd36242" containerID="a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9" exitCode=0 Sep 10 15:47:28.631370 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:28.630997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" event={"ID":"5a34a398-bd73-4a82-b402-a2883fd36242","Type":"ContainerDied","Data":"a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9"} Sep 10 15:47:28.631451 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:28.631403 2580 scope.go:117] "RemoveContainer" containerID="a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9" Sep 10 15:47:28.632475 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:28.632445 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:47:28.632528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:28.632511 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:47:28.632569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:28.632541 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:47:29.641973 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:29.641927 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" event={"ID":"5a34a398-bd73-4a82-b402-a2883fd36242","Type":"ContainerStarted","Data":"3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78"} Sep 10 15:47:29.995254 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:29.995174 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:29.995410 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:29.995350 2580 patch_prober.go:28] interesting pod/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:47:29.995410 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:29.995397 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:47:30.995744 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:30.995707 2580 patch_prober.go:28] interesting pod/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:47:30.996238 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:30.995767 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:47:31.703471 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.703435 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6"] Sep 10 15:47:31.703835 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.703823 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76856e53-f04a-4f25-aea5-b6a4b9464a1d" containerName="revoke-keys" Sep 10 15:47:31.703878 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.703836 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="76856e53-f04a-4f25-aea5-b6a4b9464a1d" containerName="revoke-keys" Sep 10 15:47:31.703878 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.703854 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" Sep 10 15:47:31.703878 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.703859 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bfd4b0e-0334-4c62-be04-579a38364573" containerName="istio-proxy" Sep 10 15:47:31.703968 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.703935 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="76856e53-f04a-4f25-aea5-b6a4b9464a1d" containerName="revoke-keys" Sep 10 15:47:31.706348 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.706332 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" Sep 10 15:47:31.708608 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.708583 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:47:31.708740 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.708658 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:47:31.714298 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.714271 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6"] Sep 10 15:47:31.824447 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.822025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81de020a-2cf3-4a04-983d-4e5fee934e6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6\" (UID: \"81de020a-2cf3-4a04-983d-4e5fee934e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" Sep 10 15:47:31.923830 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.923783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81de020a-2cf3-4a04-983d-4e5fee934e6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6\" (UID: \"81de020a-2cf3-4a04-983d-4e5fee934e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" Sep 10 15:47:31.926359 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.925527 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81de020a-2cf3-4a04-983d-4e5fee934e6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6\" (UID: \"81de020a-2cf3-4a04-983d-4e5fee934e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" Sep 10 15:47:31.995824 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.995738 2580 patch_prober.go:28] interesting pod/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:47:31.995824 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:31.995794 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:47:32.019696 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:32.019662 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" Sep 10 15:47:32.307980 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:32.307951 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6"] Sep 10 15:47:32.309807 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:47:32.309779 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81de020a_2cf3_4a04_983d_4e5fee934e6f.slice/crio-629786ff65ec55da4b0e6d68f7bb25cd8b7e5fd750acb7a719ed3372d4c9812d WatchSource:0}: Error finding container 629786ff65ec55da4b0e6d68f7bb25cd8b7e5fd750acb7a719ed3372d4c9812d: Status 404 returned error can't find the container with id 629786ff65ec55da4b0e6d68f7bb25cd8b7e5fd750acb7a719ed3372d4c9812d Sep 10 15:47:32.659399 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:32.659367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" event={"ID":"81de020a-2cf3-4a04-983d-4e5fee934e6f","Type":"ContainerStarted","Data":"629786ff65ec55da4b0e6d68f7bb25cd8b7e5fd750acb7a719ed3372d4c9812d"} Sep 10 15:47:32.995663 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:32.995578 2580 patch_prober.go:28] interesting pod/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:47:32.995663 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:32.995637 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:47:33.666163 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:33.666125 2580 generic.go:356] "Generic (PLEG): container finished" podID="81de020a-2cf3-4a04-983d-4e5fee934e6f" containerID="49d274446277c5e553a880f9a056c99103ed37204055113b1af367516f56d2cb" exitCode=0 Sep 10 15:47:33.666631 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:33.666219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" event={"ID":"81de020a-2cf3-4a04-983d-4e5fee934e6f","Type":"ContainerDied","Data":"49d274446277c5e553a880f9a056c99103ed37204055113b1af367516f56d2cb"} Sep 10 15:47:33.994915 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:33.994844 2580 patch_prober.go:28] interesting pod/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 10 15:47:33.994915 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:33.994900 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 10 15:47:34.041047 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:34.041015 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2"] Sep 10 15:47:34.045621 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:34.045595 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d5048f-sp6m2"] Sep 10 15:47:34.816962 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:34.816941 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" Sep 10 15:47:34.856708 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:34.856675 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81de020a-2cf3-4a04-983d-4e5fee934e6f-maas-api-service-ca\") pod \"81de020a-2cf3-4a04-983d-4e5fee934e6f\" (UID: \"81de020a-2cf3-4a04-983d-4e5fee934e6f\") " Sep 10 15:47:34.857114 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:34.857083 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81de020a-2cf3-4a04-983d-4e5fee934e6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81de020a-2cf3-4a04-983d-4e5fee934e6f" (UID: "81de020a-2cf3-4a04-983d-4e5fee934e6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:47:34.958587 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:34.958504 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81de020a-2cf3-4a04-983d-4e5fee934e6f-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:34.999527 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:34.999499 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:34.999785 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:34.999765 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:35.000470 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:35.000418 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:35.679718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:35.679691 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" Sep 10 15:47:35.679869 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:35.679733 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6" event={"ID":"81de020a-2cf3-4a04-983d-4e5fee934e6f","Type":"ContainerDied","Data":"629786ff65ec55da4b0e6d68f7bb25cd8b7e5fd750acb7a719ed3372d4c9812d"} Sep 10 15:47:35.679869 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:35.679775 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="629786ff65ec55da4b0e6d68f7bb25cd8b7e5fd750acb7a719ed3372d4c9812d" Sep 10 15:47:36.029177 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:36.029102 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76856e53-f04a-4f25-aea5-b6a4b9464a1d" path="/var/lib/kubelet/pods/76856e53-f04a-4f25-aea5-b6a4b9464a1d/volumes" Sep 10 15:47:39.153099 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:39.153057 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6"] Sep 10 15:47:39.155169 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:39.155147 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ff2a412a-ctkb6"] Sep 10 15:47:39.290333 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:39.290296 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6"] Sep 10 15:47:39.290558 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:39.290517 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" containerID="cri-o://3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78" gracePeriod=30 Sep 10 15:47:40.030043 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:40.030006 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81de020a-2cf3-4a04-983d-4e5fee934e6f" path="/var/lib/kubelet/pods/81de020a-2cf3-4a04-983d-4e5fee934e6f/volumes" Sep 10 15:47:44.547371 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.547346 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:44.660298 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660270 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-certs\") pod \"5a34a398-bd73-4a82-b402-a2883fd36242\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " Sep 10 15:47:44.660501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660318 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34a398-bd73-4a82-b402-a2883fd36242-istiod-ca-cert\") pod \"5a34a398-bd73-4a82-b402-a2883fd36242\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " Sep 10 15:47:44.660501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660339 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-envoy\") pod \"5a34a398-bd73-4a82-b402-a2883fd36242\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " Sep 10 15:47:44.660501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660399 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv5tq\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-kube-api-access-xv5tq\") pod \"5a34a398-bd73-4a82-b402-a2883fd36242\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " Sep 10 15:47:44.660501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660443 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34a398-bd73-4a82-b402-a2883fd36242-istio-podinfo\") pod \"5a34a398-bd73-4a82-b402-a2883fd36242\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " Sep 10 15:47:44.660501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660474 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-socket\") pod \"5a34a398-bd73-4a82-b402-a2883fd36242\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " Sep 10 15:47:44.660501 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660502 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-data\") pod \"5a34a398-bd73-4a82-b402-a2883fd36242\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " Sep 10 15:47:44.660841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660565 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-credential-socket\") pod \"5a34a398-bd73-4a82-b402-a2883fd36242\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " Sep 10 15:47:44.660841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660580 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a34a398-bd73-4a82-b402-a2883fd36242" (UID: "5a34a398-bd73-4a82-b402-a2883fd36242"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:44.660841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660600 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-istio-token\") pod \"5a34a398-bd73-4a82-b402-a2883fd36242\" (UID: \"5a34a398-bd73-4a82-b402-a2883fd36242\") " Sep 10 15:47:44.660841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660778 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a34a398-bd73-4a82-b402-a2883fd36242" (UID: "5a34a398-bd73-4a82-b402-a2883fd36242"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:44.660841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660827 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a34a398-bd73-4a82-b402-a2883fd36242" (UID: "5a34a398-bd73-4a82-b402-a2883fd36242"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:44.660841 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660785 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a34a398-bd73-4a82-b402-a2883fd36242" (UID: "5a34a398-bd73-4a82-b402-a2883fd36242"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:44.661142 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660842 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a34a398-bd73-4a82-b402-a2883fd36242-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a34a398-bd73-4a82-b402-a2883fd36242" (UID: "5a34a398-bd73-4a82-b402-a2883fd36242"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:47:44.661142 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660968 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:44.661142 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660985 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:44.661142 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.660994 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a34a398-bd73-4a82-b402-a2883fd36242-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:44.661142 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.661005 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:44.661142 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.661016 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:44.662684 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.662659 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a34a398-bd73-4a82-b402-a2883fd36242-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a34a398-bd73-4a82-b402-a2883fd36242" (UID: "5a34a398-bd73-4a82-b402-a2883fd36242"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:47:44.662892 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.662866 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-kube-api-access-xv5tq" (OuterVolumeSpecName: "kube-api-access-xv5tq") pod "5a34a398-bd73-4a82-b402-a2883fd36242" (UID: "5a34a398-bd73-4a82-b402-a2883fd36242"). InnerVolumeSpecName "kube-api-access-xv5tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:47:44.663161 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.663147 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a34a398-bd73-4a82-b402-a2883fd36242" (UID: "5a34a398-bd73-4a82-b402-a2883fd36242"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:44.663206 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.663176 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a34a398-bd73-4a82-b402-a2883fd36242" (UID: "5a34a398-bd73-4a82-b402-a2883fd36242"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:47:44.728704 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.728670 2580 generic.go:356] "Generic (PLEG): container finished" podID="5a34a398-bd73-4a82-b402-a2883fd36242" containerID="3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78" exitCode=0 Sep 10 15:47:44.728844 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.728722 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" Sep 10 15:47:44.728844 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.728759 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" event={"ID":"5a34a398-bd73-4a82-b402-a2883fd36242","Type":"ContainerDied","Data":"3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78"} Sep 10 15:47:44.728844 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.728809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6" event={"ID":"5a34a398-bd73-4a82-b402-a2883fd36242","Type":"ContainerDied","Data":"33a39d8824f63042855789a4f1224a219ac35b295322f30b999d26939a0573e7"} Sep 10 15:47:44.728844 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.728833 2580 scope.go:117] "RemoveContainer" containerID="3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78" Sep 10 15:47:44.737799 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.737780 2580 scope.go:117] "RemoveContainer" containerID="a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9" Sep 10 15:47:44.745714 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.745697 2580 scope.go:117] "RemoveContainer" containerID="3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78" Sep 10 15:47:44.745970 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:47:44.745953 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78\": container with ID starting with 3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78 not found: ID does not exist" containerID="3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78" Sep 10 15:47:44.746019 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.745977 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78"} err="failed to get container status \"3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78\": rpc error: code = NotFound desc = could not find container \"3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78\": container with ID starting with 3b03b39ee1c4715afe38283a0a209c587cb09feab2ae6bbb2e12fd9506b4bd78 not found: ID does not exist" Sep 10 15:47:44.746019 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.745991 2580 scope.go:117] "RemoveContainer" containerID="a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9" Sep 10 15:47:44.746238 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:47:44.746220 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9\": container with ID starting with a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9 not found: ID does not exist" containerID="a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9" Sep 10 15:47:44.746288 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.746240 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9"} err="failed to get container status \"a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9\": rpc error: code = NotFound desc = could not find container \"a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9\": container with ID starting with a5579762754923ed25f97d962549db4661a3195754e8902284e3f430993ad1c9 not found: ID does not exist" Sep 10 15:47:44.749718 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.749697 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6"] Sep 10 15:47:44.753141 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.753122 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ff2a412a-openshift-default-6968688b5b-vntd6"] Sep 10 15:47:44.762102 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.762078 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:44.762102 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.762098 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a34a398-bd73-4a82-b402-a2883fd36242-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:44.762225 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.762109 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv5tq\" (UniqueName: \"kubernetes.io/projected/5a34a398-bd73-4a82-b402-a2883fd36242-kube-api-access-xv5tq\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:44.762225 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:44.762118 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a34a398-bd73-4a82-b402-a2883fd36242-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:47:46.028533 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:47:46.028504 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" path="/var/lib/kubelet/pods/5a34a398-bd73-4a82-b402-a2883fd36242/volumes" Sep 10 15:48:16.466703 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:16.466671 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z"] Sep 10 15:48:16.467280 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:16.466902 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" containerID="cri-o://b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10" gracePeriod=30 Sep 10 15:48:21.736713 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.736647 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:48:21.837462 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837403 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-data\") pod \"d73d988b-e54c-4552-8fec-f9890599c92f\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " Sep 10 15:48:21.837462 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837459 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-envoy\") pod \"d73d988b-e54c-4552-8fec-f9890599c92f\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " Sep 10 15:48:21.837720 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837510 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-socket\") pod \"d73d988b-e54c-4552-8fec-f9890599c92f\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " Sep 10 15:48:21.837720 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837544 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d988b-e54c-4552-8fec-f9890599c92f-istio-podinfo\") pod \"d73d988b-e54c-4552-8fec-f9890599c92f\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " Sep 10 15:48:21.837720 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837563 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-certs\") pod \"d73d988b-e54c-4552-8fec-f9890599c92f\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " Sep 10 15:48:21.837867 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837730 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-data" (OuterVolumeSpecName: "istio-data") pod "d73d988b-e54c-4552-8fec-f9890599c92f" (UID: "d73d988b-e54c-4552-8fec-f9890599c92f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:21.837867 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837749 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-credential-socket\") pod \"d73d988b-e54c-4552-8fec-f9890599c92f\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " Sep 10 15:48:21.837867 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837824 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d73d988b-e54c-4552-8fec-f9890599c92f" (UID: "d73d988b-e54c-4552-8fec-f9890599c92f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:21.837867 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837828 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d988b-e54c-4552-8fec-f9890599c92f-istiod-ca-cert\") pod \"d73d988b-e54c-4552-8fec-f9890599c92f\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " Sep 10 15:48:21.838083 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837839 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d73d988b-e54c-4552-8fec-f9890599c92f" (UID: "d73d988b-e54c-4552-8fec-f9890599c92f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:21.838083 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837874 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-istio-token\") pod \"d73d988b-e54c-4552-8fec-f9890599c92f\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " Sep 10 15:48:21.838083 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837929 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfhzx\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-kube-api-access-cfhzx\") pod \"d73d988b-e54c-4552-8fec-f9890599c92f\" (UID: \"d73d988b-e54c-4552-8fec-f9890599c92f\") " Sep 10 15:48:21.838083 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.837979 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d73d988b-e54c-4552-8fec-f9890599c92f" (UID: "d73d988b-e54c-4552-8fec-f9890599c92f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:21.838256 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.838098 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73d988b-e54c-4552-8fec-f9890599c92f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d73d988b-e54c-4552-8fec-f9890599c92f" (UID: "d73d988b-e54c-4552-8fec-f9890599c92f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:48:21.838298 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.838253 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:21.838298 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.838279 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:21.838298 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.838294 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:21.838413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.838305 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:21.838413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.838323 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d73d988b-e54c-4552-8fec-f9890599c92f-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:21.839916 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.839883 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d73d988b-e54c-4552-8fec-f9890599c92f" (UID: "d73d988b-e54c-4552-8fec-f9890599c92f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:21.840019 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.839918 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-kube-api-access-cfhzx" (OuterVolumeSpecName: "kube-api-access-cfhzx") pod "d73d988b-e54c-4552-8fec-f9890599c92f" (UID: "d73d988b-e54c-4552-8fec-f9890599c92f"). InnerVolumeSpecName "kube-api-access-cfhzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:48:21.840140 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.840119 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d73d988b-e54c-4552-8fec-f9890599c92f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d73d988b-e54c-4552-8fec-f9890599c92f" (UID: "d73d988b-e54c-4552-8fec-f9890599c92f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:48:21.840198 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.840158 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d73d988b-e54c-4552-8fec-f9890599c92f" (UID: "d73d988b-e54c-4552-8fec-f9890599c92f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:48:21.929439 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.929398 2580 generic.go:356] "Generic (PLEG): container finished" podID="d73d988b-e54c-4552-8fec-f9890599c92f" containerID="b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10" exitCode=0 Sep 10 15:48:21.929593 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.929467 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" Sep 10 15:48:21.929593 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.929481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" event={"ID":"d73d988b-e54c-4552-8fec-f9890599c92f","Type":"ContainerDied","Data":"b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10"} Sep 10 15:48:21.929593 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.929540 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z" event={"ID":"d73d988b-e54c-4552-8fec-f9890599c92f","Type":"ContainerDied","Data":"2349e4bc5df3773e5ee5a333b96ac2aa7ee286bf38341ca211f675b5e4fd5e90"} Sep 10 15:48:21.929593 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.929566 2580 scope.go:117] "RemoveContainer" containerID="b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10" Sep 10 15:48:21.939290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.939269 2580 scope.go:117] "RemoveContainer" containerID="cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486" Sep 10 15:48:21.939763 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.939736 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhzx\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-kube-api-access-cfhzx\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:21.939763 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.939762 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d73d988b-e54c-4552-8fec-f9890599c92f-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:21.939908 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.939777 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d73d988b-e54c-4552-8fec-f9890599c92f-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:21.939908 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.939789 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d73d988b-e54c-4552-8fec-f9890599c92f-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:21.948157 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.948130 2580 scope.go:117] "RemoveContainer" containerID="b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10" Sep 10 15:48:21.948692 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:48:21.948605 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10\": container with ID starting with b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10 not found: ID does not exist" containerID="b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10" Sep 10 15:48:21.948692 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.948657 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10"} err="failed to get container status \"b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10\": rpc error: code = NotFound desc = could not find container \"b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10\": container with ID starting with b9dd2e5c65f0935a4822277c049a7c40323798db2ae1114ad0c6894b8f7a2a10 not found: ID does not exist" Sep 10 15:48:21.948692 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.948680 2580 scope.go:117] "RemoveContainer" containerID="cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486" Sep 10 15:48:21.949137 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:48:21.949062 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486\": container with ID starting with cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486 not found: ID does not exist" containerID="cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486" Sep 10 15:48:21.949137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.949101 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486"} err="failed to get container status \"cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486\": rpc error: code = NotFound desc = could not find container \"cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486\": container with ID starting with cb77d58a4f12c2e16516d65452a7ff0cab2571b4cd0997cc070805b15dc2f486 not found: ID does not exist" Sep 10 15:48:21.952406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.952380 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z"] Sep 10 15:48:21.961181 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:21.959098 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-d5048f-openshift-default-5546cb67b4-tcf2z"] Sep 10 15:48:22.029791 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.029712 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" path="/var/lib/kubelet/pods/d73d988b-e54c-4552-8fec-f9890599c92f/volumes" Sep 10 15:48:22.248320 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248285 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54"] Sep 10 15:48:22.248722 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248708 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" Sep 10 15:48:22.248770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248724 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" Sep 10 15:48:22.248770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248735 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" Sep 10 15:48:22.248770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248742 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" Sep 10 15:48:22.248770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248754 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" Sep 10 15:48:22.248770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248762 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" Sep 10 15:48:22.248923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248782 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" Sep 10 15:48:22.248923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248788 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" Sep 10 15:48:22.248923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248797 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81de020a-2cf3-4a04-983d-4e5fee934e6f" containerName="revoke-keys" Sep 10 15:48:22.248923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248802 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="81de020a-2cf3-4a04-983d-4e5fee934e6f" containerName="revoke-keys" Sep 10 15:48:22.248923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248865 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" Sep 10 15:48:22.248923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248873 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" Sep 10 15:48:22.248923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248881 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a34a398-bd73-4a82-b402-a2883fd36242" containerName="istio-proxy" Sep 10 15:48:22.248923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.248887 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="81de020a-2cf3-4a04-983d-4e5fee934e6f" containerName="revoke-keys" Sep 10 15:48:22.254193 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.254175 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:22.257933 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.257915 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-76a9b8\"" Sep 10 15:48:22.257933 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.257927 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-76a9b8-dockercfg-pzz2r\"" Sep 10 15:48:22.270608 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.270581 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54"] Sep 10 15:48:22.344843 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.344452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8smvh\" (UniqueName: \"kubernetes.io/projected/f75dda8c-979c-4f6e-97d2-791412ee4afe-kube-api-access-8smvh\") pod \"payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54\" (UID: \"f75dda8c-979c-4f6e-97d2-791412ee4afe\") " pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:22.344843 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.344581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75dda8c-979c-4f6e-97d2-791412ee4afe-plugins-config-volume\") pod \"payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54\" (UID: \"f75dda8c-979c-4f6e-97d2-791412ee4afe\") " pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:22.445440 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.445395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8smvh\" (UniqueName: \"kubernetes.io/projected/f75dda8c-979c-4f6e-97d2-791412ee4afe-kube-api-access-8smvh\") pod \"payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54\" (UID: \"f75dda8c-979c-4f6e-97d2-791412ee4afe\") " pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:22.445638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.445507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75dda8c-979c-4f6e-97d2-791412ee4afe-plugins-config-volume\") pod \"payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54\" (UID: \"f75dda8c-979c-4f6e-97d2-791412ee4afe\") " pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:22.446145 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.446122 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75dda8c-979c-4f6e-97d2-791412ee4afe-plugins-config-volume\") pod \"payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54\" (UID: \"f75dda8c-979c-4f6e-97d2-791412ee4afe\") " pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:22.454762 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.454734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8smvh\" (UniqueName: \"kubernetes.io/projected/f75dda8c-979c-4f6e-97d2-791412ee4afe-kube-api-access-8smvh\") pod \"payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54\" (UID: \"f75dda8c-979c-4f6e-97d2-791412ee4afe\") " pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:22.566033 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.565994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:22.839386 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.839358 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54"] Sep 10 15:48:22.841295 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:48:22.841258 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf75dda8c_979c_4f6e_97d2_791412ee4afe.slice/crio-7c2accc0d5d720d3ac649148f61788f987c6e03fc5f376102ae9168560de43f9 WatchSource:0}: Error finding container 7c2accc0d5d720d3ac649148f61788f987c6e03fc5f376102ae9168560de43f9: Status 404 returned error can't find the container with id 7c2accc0d5d720d3ac649148f61788f987c6e03fc5f376102ae9168560de43f9 Sep 10 15:48:22.938311 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.938282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" event={"ID":"f75dda8c-979c-4f6e-97d2-791412ee4afe","Type":"ContainerStarted","Data":"6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995"} Sep 10 15:48:22.938483 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.938327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" event={"ID":"f75dda8c-979c-4f6e-97d2-791412ee4afe","Type":"ContainerStarted","Data":"7c2accc0d5d720d3ac649148f61788f987c6e03fc5f376102ae9168560de43f9"} Sep 10 15:48:22.938483 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.938391 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:22.957629 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:22.957578 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" podStartSLOduration=0.957560999 podStartE2EDuration="957.560999ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:22.955306346 +0000 UTC m=+961.517615536" watchObservedRunningTime="2026-09-10 15:48:22.957560999 +0000 UTC m=+961.519870189" Sep 10 15:48:33.946034 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:33.945996 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:48:52.756668 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:52.756590 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2"] Sep 10 15:48:52.757334 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:52.757313 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d73d988b-e54c-4552-8fec-f9890599c92f" containerName="istio-proxy" Sep 10 15:48:52.760757 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:52.760740 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" Sep 10 15:48:52.762961 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:52.762940 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:48:52.763340 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:52.763069 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:48:52.765260 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:52.765240 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2"] Sep 10 15:48:52.847044 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:52.847012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8619b9cd-7d38-410a-92f4-a12aca7c4053-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-76a9b8-674d2\" (UID: \"8619b9cd-7d38-410a-92f4-a12aca7c4053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" Sep 10 15:48:52.948117 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:52.948083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8619b9cd-7d38-410a-92f4-a12aca7c4053-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-76a9b8-674d2\" (UID: \"8619b9cd-7d38-410a-92f4-a12aca7c4053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" Sep 10 15:48:52.948743 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:52.948722 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8619b9cd-7d38-410a-92f4-a12aca7c4053-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-76a9b8-674d2\" (UID: \"8619b9cd-7d38-410a-92f4-a12aca7c4053\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" Sep 10 15:48:53.075283 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:53.075197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" Sep 10 15:48:53.346300 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:53.346276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2"] Sep 10 15:48:53.348085 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:48:53.348059 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8619b9cd_7d38_410a_92f4_a12aca7c4053.slice/crio-72962075e48df02cc400168d178e413b8acbd37cb91456958d72e34af1b02253 WatchSource:0}: Error finding container 72962075e48df02cc400168d178e413b8acbd37cb91456958d72e34af1b02253: Status 404 returned error can't find the container with id 72962075e48df02cc400168d178e413b8acbd37cb91456958d72e34af1b02253 Sep 10 15:48:54.109011 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:54.108975 2580 generic.go:356] "Generic (PLEG): container finished" podID="8619b9cd-7d38-410a-92f4-a12aca7c4053" containerID="7765902f672e995227fabb58c3fe32fefbdadb1a88a0e0d982d945b77f51b30f" exitCode=0 Sep 10 15:48:54.109381 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:54.109058 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" event={"ID":"8619b9cd-7d38-410a-92f4-a12aca7c4053","Type":"ContainerDied","Data":"7765902f672e995227fabb58c3fe32fefbdadb1a88a0e0d982d945b77f51b30f"} Sep 10 15:48:54.109381 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:54.109106 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" event={"ID":"8619b9cd-7d38-410a-92f4-a12aca7c4053","Type":"ContainerStarted","Data":"72962075e48df02cc400168d178e413b8acbd37cb91456958d72e34af1b02253"} Sep 10 15:48:55.247580 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:55.247554 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" Sep 10 15:48:55.372897 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:55.372798 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8619b9cd-7d38-410a-92f4-a12aca7c4053-maas-api-service-ca\") pod \"8619b9cd-7d38-410a-92f4-a12aca7c4053\" (UID: \"8619b9cd-7d38-410a-92f4-a12aca7c4053\") " Sep 10 15:48:55.373154 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:55.373130 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8619b9cd-7d38-410a-92f4-a12aca7c4053-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8619b9cd-7d38-410a-92f4-a12aca7c4053" (UID: "8619b9cd-7d38-410a-92f4-a12aca7c4053"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:48:55.474443 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:55.474388 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8619b9cd-7d38-410a-92f4-a12aca7c4053-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:48:56.120840 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:56.120802 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" event={"ID":"8619b9cd-7d38-410a-92f4-a12aca7c4053","Type":"ContainerDied","Data":"72962075e48df02cc400168d178e413b8acbd37cb91456958d72e34af1b02253"} Sep 10 15:48:56.120840 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:56.120842 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2" Sep 10 15:48:56.121042 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:48:56.120859 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72962075e48df02cc400168d178e413b8acbd37cb91456958d72e34af1b02253" Sep 10 15:49:03.208289 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.208252 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54"] Sep 10 15:49:03.208772 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.208554 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" podUID="f75dda8c-979c-4f6e-97d2-791412ee4afe" containerName="payload-processing" containerID="cri-o://6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995" gracePeriod=30 Sep 10 15:49:03.369383 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.369353 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2"] Sep 10 15:49:03.372364 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.372332 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-76a9b8-674d2"] Sep 10 15:49:03.462193 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.462128 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:49:03.551222 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.551174 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75dda8c-979c-4f6e-97d2-791412ee4afe-plugins-config-volume\") pod \"f75dda8c-979c-4f6e-97d2-791412ee4afe\" (UID: \"f75dda8c-979c-4f6e-97d2-791412ee4afe\") " Sep 10 15:49:03.551444 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.551238 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8smvh\" (UniqueName: \"kubernetes.io/projected/f75dda8c-979c-4f6e-97d2-791412ee4afe-kube-api-access-8smvh\") pod \"f75dda8c-979c-4f6e-97d2-791412ee4afe\" (UID: \"f75dda8c-979c-4f6e-97d2-791412ee4afe\") " Sep 10 15:49:03.551996 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.551967 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f75dda8c-979c-4f6e-97d2-791412ee4afe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f75dda8c-979c-4f6e-97d2-791412ee4afe" (UID: "f75dda8c-979c-4f6e-97d2-791412ee4afe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:49:03.553592 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.553514 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f75dda8c-979c-4f6e-97d2-791412ee4afe-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:49:03.555400 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.555369 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f75dda8c-979c-4f6e-97d2-791412ee4afe-kube-api-access-8smvh" (OuterVolumeSpecName: "kube-api-access-8smvh") pod "f75dda8c-979c-4f6e-97d2-791412ee4afe" (UID: "f75dda8c-979c-4f6e-97d2-791412ee4afe"). InnerVolumeSpecName "kube-api-access-8smvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:49:03.654871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:03.654829 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8smvh\" (UniqueName: \"kubernetes.io/projected/f75dda8c-979c-4f6e-97d2-791412ee4afe-kube-api-access-8smvh\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:49:04.028629 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.028590 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8619b9cd-7d38-410a-92f4-a12aca7c4053" path="/var/lib/kubelet/pods/8619b9cd-7d38-410a-92f4-a12aca7c4053/volumes" Sep 10 15:49:04.165173 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.165139 2580 generic.go:356] "Generic (PLEG): container finished" podID="f75dda8c-979c-4f6e-97d2-791412ee4afe" containerID="6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995" exitCode=0 Sep 10 15:49:04.165361 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.165178 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" Sep 10 15:49:04.165361 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.165226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" event={"ID":"f75dda8c-979c-4f6e-97d2-791412ee4afe","Type":"ContainerDied","Data":"6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995"} Sep 10 15:49:04.165361 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.165282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54" event={"ID":"f75dda8c-979c-4f6e-97d2-791412ee4afe","Type":"ContainerDied","Data":"7c2accc0d5d720d3ac649148f61788f987c6e03fc5f376102ae9168560de43f9"} Sep 10 15:49:04.165361 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.165341 2580 scope.go:117] "RemoveContainer" containerID="6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995" Sep 10 15:49:04.175264 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.175238 2580 scope.go:117] "RemoveContainer" containerID="6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995" Sep 10 15:49:04.175546 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:49:04.175526 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995\": container with ID starting with 6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995 not found: ID does not exist" containerID="6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995" Sep 10 15:49:04.175611 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.175555 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995"} err="failed to get container status \"6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995\": rpc error: code = NotFound desc = could not find container \"6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995\": container with ID starting with 6baee869d53a00900b115c5d5a0f647e30a4c925648fae3b9268b5bd96ccf995 not found: ID does not exist" Sep 10 15:49:04.181972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.181946 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54"] Sep 10 15:49:04.183950 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:04.183931 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-76a9b8-94c9cc9df-hqd54"] Sep 10 15:49:06.028075 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:06.028043 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f75dda8c-979c-4f6e-97d2-791412ee4afe" path="/var/lib/kubelet/pods/f75dda8c-979c-4f6e-97d2-791412ee4afe/volumes" Sep 10 15:49:15.797276 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.797240 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v"] Sep 10 15:49:15.797796 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.797777 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f75dda8c-979c-4f6e-97d2-791412ee4afe" containerName="payload-processing" Sep 10 15:49:15.797865 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.797798 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f75dda8c-979c-4f6e-97d2-791412ee4afe" containerName="payload-processing" Sep 10 15:49:15.797865 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.797814 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8619b9cd-7d38-410a-92f4-a12aca7c4053" containerName="revoke-keys" Sep 10 15:49:15.797865 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.797819 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8619b9cd-7d38-410a-92f4-a12aca7c4053" containerName="revoke-keys" Sep 10 15:49:15.798026 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.797885 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f75dda8c-979c-4f6e-97d2-791412ee4afe" containerName="payload-processing" Sep 10 15:49:15.798026 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.797895 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8619b9cd-7d38-410a-92f4-a12aca7c4053" containerName="revoke-keys" Sep 10 15:49:15.802438 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.802393 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:15.804947 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.804923 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-f32d75-openshift-default-dockercfg-vn2hr\"" Sep 10 15:49:15.815310 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.815286 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v"] Sep 10 15:49:15.976396 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.976356 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78kvl\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-kube-api-access-78kvl\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:15.976608 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.976449 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-credential-socket\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:15.976608 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.976480 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-socket\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:15.976608 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.976516 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-data\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:15.976780 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.976605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-podinfo\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:15.976780 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.976639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-token\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:15.976780 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.976668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-envoy\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:15.976780 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.976696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-certs\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:15.976780 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:15.976730 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919956cb-9dfc-47f1-99ce-26222cd8c24a-istiod-ca-cert\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.077559 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-credential-socket\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.077559 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-socket\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.077559 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-data\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.077832 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-podinfo\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.077832 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-token\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.077832 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-envoy\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.077832 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-certs\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.077832 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919956cb-9dfc-47f1-99ce-26222cd8c24a-istiod-ca-cert\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.077832 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78kvl\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-kube-api-access-78kvl\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.078153 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077947 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-socket\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.078153 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.077966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-credential-socket\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.078272 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.078202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-certs\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.078272 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.078209 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-data\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.078522 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.078503 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919956cb-9dfc-47f1-99ce-26222cd8c24a-istiod-ca-cert\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.080006 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.079979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-envoy\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.080210 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.080192 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-podinfo\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.085121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.085097 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78kvl\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-kube-api-access-78kvl\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.085215 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.085103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-token\") pod \"e2e-noten-f32d75-openshift-default-856585647d-xhm8v\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.115165 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.115144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:16.394148 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.394120 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v"] Sep 10 15:49:16.395487 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:49:16.395460 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919956cb_9dfc_47f1_99ce_26222cd8c24a.slice/crio-246301a0b864c3ba62b10d185eefee4c9d817471b21878408624b5e1e5a2b846 WatchSource:0}: Error finding container 246301a0b864c3ba62b10d185eefee4c9d817471b21878408624b5e1e5a2b846: Status 404 returned error can't find the container with id 246301a0b864c3ba62b10d185eefee4c9d817471b21878408624b5e1e5a2b846 Sep 10 15:49:16.397687 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.397655 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:49:16.397784 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.397734 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:49:16.397784 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:16.397764 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:49:17.238090 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:17.238050 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" event={"ID":"919956cb-9dfc-47f1-99ce-26222cd8c24a","Type":"ContainerStarted","Data":"5a07a3b42a45f7911b077265c4ac1a4ad8ef311bd44ce01429be5e78d7b9afe8"} Sep 10 15:49:17.238515 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:17.238119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" event={"ID":"919956cb-9dfc-47f1-99ce-26222cd8c24a","Type":"ContainerStarted","Data":"246301a0b864c3ba62b10d185eefee4c9d817471b21878408624b5e1e5a2b846"} Sep 10 15:49:17.258195 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:17.258135 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" podStartSLOduration=2.258114023 podStartE2EDuration="2.258114023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:17.257210032 +0000 UTC m=+1015.819519225" watchObservedRunningTime="2026-09-10 15:49:17.258114023 +0000 UTC m=+1015.820423214" Sep 10 15:49:18.115717 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:18.115686 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:18.121169 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:18.121146 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:18.243072 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:18.243040 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:18.243959 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:18.243942 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:49:50.218054 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.218009 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl"] Sep 10 15:49:50.221715 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.221695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" Sep 10 15:49:50.224232 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.224210 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:49:50.224338 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.224210 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:49:50.231040 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.231016 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl"] Sep 10 15:49:50.321573 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.321533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19170c79-f95a-4608-a1a0-b8e35fdd5627-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25381b71-t6gfl\" (UID: \"19170c79-f95a-4608-a1a0-b8e35fdd5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" Sep 10 15:49:50.422763 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.422735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19170c79-f95a-4608-a1a0-b8e35fdd5627-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25381b71-t6gfl\" (UID: \"19170c79-f95a-4608-a1a0-b8e35fdd5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" Sep 10 15:49:50.423355 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.423336 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19170c79-f95a-4608-a1a0-b8e35fdd5627-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25381b71-t6gfl\" (UID: \"19170c79-f95a-4608-a1a0-b8e35fdd5627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" Sep 10 15:49:50.533692 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.533612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" Sep 10 15:49:50.806646 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:50.806622 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl"] Sep 10 15:49:50.808690 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:49:50.808663 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19170c79_f95a_4608_a1a0_b8e35fdd5627.slice/crio-acd7f0375dbb87eedb8c1abd0d8957b6ddd921d95e77e2e42497339d1f8334f1 WatchSource:0}: Error finding container acd7f0375dbb87eedb8c1abd0d8957b6ddd921d95e77e2e42497339d1f8334f1: Status 404 returned error can't find the container with id acd7f0375dbb87eedb8c1abd0d8957b6ddd921d95e77e2e42497339d1f8334f1 Sep 10 15:49:51.423035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:51.422869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" event={"ID":"19170c79-f95a-4608-a1a0-b8e35fdd5627","Type":"ContainerStarted","Data":"5f607de9fbccae484c91f181f61f7a141a928fa3fc013ae2a73a61556458e370"} Sep 10 15:49:51.423035 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:51.423021 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" event={"ID":"19170c79-f95a-4608-a1a0-b8e35fdd5627","Type":"ContainerStarted","Data":"acd7f0375dbb87eedb8c1abd0d8957b6ddd921d95e77e2e42497339d1f8334f1"} Sep 10 15:49:51.438607 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:51.438557 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" podStartSLOduration=1.438537995 podStartE2EDuration="1.438537995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:51.437077287 +0000 UTC m=+1049.999386479" watchObservedRunningTime="2026-09-10 15:49:51.438537995 +0000 UTC m=+1050.000847186" Sep 10 15:49:52.428845 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:52.428809 2580 generic.go:356] "Generic (PLEG): container finished" podID="19170c79-f95a-4608-a1a0-b8e35fdd5627" containerID="5f607de9fbccae484c91f181f61f7a141a928fa3fc013ae2a73a61556458e370" exitCode=0 Sep 10 15:49:52.429275 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:52.428858 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" event={"ID":"19170c79-f95a-4608-a1a0-b8e35fdd5627","Type":"ContainerDied","Data":"5f607de9fbccae484c91f181f61f7a141a928fa3fc013ae2a73a61556458e370"} Sep 10 15:49:53.569284 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:53.569254 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" Sep 10 15:49:53.757225 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:53.757153 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19170c79-f95a-4608-a1a0-b8e35fdd5627-maas-api-service-ca\") pod \"19170c79-f95a-4608-a1a0-b8e35fdd5627\" (UID: \"19170c79-f95a-4608-a1a0-b8e35fdd5627\") " Sep 10 15:49:53.757508 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:53.757485 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19170c79-f95a-4608-a1a0-b8e35fdd5627-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19170c79-f95a-4608-a1a0-b8e35fdd5627" (UID: "19170c79-f95a-4608-a1a0-b8e35fdd5627"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:49:53.858893 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:53.858863 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19170c79-f95a-4608-a1a0-b8e35fdd5627-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:49:54.441500 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:54.441465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" event={"ID":"19170c79-f95a-4608-a1a0-b8e35fdd5627","Type":"ContainerDied","Data":"acd7f0375dbb87eedb8c1abd0d8957b6ddd921d95e77e2e42497339d1f8334f1"} Sep 10 15:49:54.441675 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:54.441476 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl" Sep 10 15:49:54.441675 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:54.441521 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd7f0375dbb87eedb8c1abd0d8957b6ddd921d95e77e2e42497339d1f8334f1" Sep 10 15:49:55.345883 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:55.345842 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v"] Sep 10 15:49:55.346519 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:55.346038 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" podUID="919956cb-9dfc-47f1-99ce-26222cd8c24a" containerName="istio-proxy" containerID="cri-o://5a07a3b42a45f7911b077265c4ac1a4ad8ef311bd44ce01429be5e78d7b9afe8" gracePeriod=30 Sep 10 15:49:56.264657 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.264617 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv"] Sep 10 15:49:56.265083 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.265069 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19170c79-f95a-4608-a1a0-b8e35fdd5627" containerName="revoke-keys" Sep 10 15:49:56.265128 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.265084 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19170c79-f95a-4608-a1a0-b8e35fdd5627" containerName="revoke-keys" Sep 10 15:49:56.265178 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.265168 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19170c79-f95a-4608-a1a0-b8e35fdd5627" containerName="revoke-keys" Sep 10 15:49:56.268477 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.268456 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" Sep 10 15:49:56.270917 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.270895 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:49:56.271122 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.271100 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:49:56.274782 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.274756 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv"] Sep 10 15:49:56.281486 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.281386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfab07f2-78e0-4073-98f3-b804d672ffa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv\" (UID: \"bfab07f2-78e0-4073-98f3-b804d672ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" Sep 10 15:49:56.382321 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.382286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfab07f2-78e0-4073-98f3-b804d672ffa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv\" (UID: \"bfab07f2-78e0-4073-98f3-b804d672ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" Sep 10 15:49:56.382924 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.382902 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfab07f2-78e0-4073-98f3-b804d672ffa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv\" (UID: \"bfab07f2-78e0-4073-98f3-b804d672ffa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" Sep 10 15:49:56.584474 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.584115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" Sep 10 15:49:56.860473 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:56.860446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv"] Sep 10 15:49:56.862690 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:49:56.862665 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfab07f2_78e0_4073_98f3_b804d672ffa8.slice/crio-563a471dd50b31529d133718410402d9dd70f7af9696d94b77c5f6e8dfc7c16c WatchSource:0}: Error finding container 563a471dd50b31529d133718410402d9dd70f7af9696d94b77c5f6e8dfc7c16c: Status 404 returned error can't find the container with id 563a471dd50b31529d133718410402d9dd70f7af9696d94b77c5f6e8dfc7c16c Sep 10 15:49:57.466572 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:57.466480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" event={"ID":"bfab07f2-78e0-4073-98f3-b804d672ffa8","Type":"ContainerStarted","Data":"510244a10c5131da33522f2808a8525af2fb38a7db818b3a99dc2bb2a8db52eb"} Sep 10 15:49:57.466572 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:57.466538 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" event={"ID":"bfab07f2-78e0-4073-98f3-b804d672ffa8","Type":"ContainerStarted","Data":"563a471dd50b31529d133718410402d9dd70f7af9696d94b77c5f6e8dfc7c16c"} Sep 10 15:49:57.482492 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:57.482441 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" podStartSLOduration=1.482415443 podStartE2EDuration="1.482415443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:57.479626969 +0000 UTC m=+1056.041936172" watchObservedRunningTime="2026-09-10 15:49:57.482415443 +0000 UTC m=+1056.044724632" Sep 10 15:49:58.473233 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:58.473199 2580 generic.go:356] "Generic (PLEG): container finished" podID="bfab07f2-78e0-4073-98f3-b804d672ffa8" containerID="510244a10c5131da33522f2808a8525af2fb38a7db818b3a99dc2bb2a8db52eb" exitCode=0 Sep 10 15:49:58.473635 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:58.473285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" event={"ID":"bfab07f2-78e0-4073-98f3-b804d672ffa8","Type":"ContainerDied","Data":"510244a10c5131da33522f2808a8525af2fb38a7db818b3a99dc2bb2a8db52eb"} Sep 10 15:49:59.623572 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:59.623542 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" Sep 10 15:49:59.714790 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:59.714760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfab07f2-78e0-4073-98f3-b804d672ffa8-maas-api-service-ca\") pod \"bfab07f2-78e0-4073-98f3-b804d672ffa8\" (UID: \"bfab07f2-78e0-4073-98f3-b804d672ffa8\") " Sep 10 15:49:59.715168 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:59.715144 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfab07f2-78e0-4073-98f3-b804d672ffa8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfab07f2-78e0-4073-98f3-b804d672ffa8" (UID: "bfab07f2-78e0-4073-98f3-b804d672ffa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:49:59.816220 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:49:59.816189 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfab07f2-78e0-4073-98f3-b804d672ffa8-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:00.485457 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.485410 2580 generic.go:356] "Generic (PLEG): container finished" podID="919956cb-9dfc-47f1-99ce-26222cd8c24a" containerID="5a07a3b42a45f7911b077265c4ac1a4ad8ef311bd44ce01429be5e78d7b9afe8" exitCode=0 Sep 10 15:50:00.485457 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.485448 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" event={"ID":"919956cb-9dfc-47f1-99ce-26222cd8c24a","Type":"ContainerDied","Data":"5a07a3b42a45f7911b077265c4ac1a4ad8ef311bd44ce01429be5e78d7b9afe8"} Sep 10 15:50:00.486929 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.486907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" event={"ID":"bfab07f2-78e0-4073-98f3-b804d672ffa8","Type":"ContainerDied","Data":"563a471dd50b31529d133718410402d9dd70f7af9696d94b77c5f6e8dfc7c16c"} Sep 10 15:50:00.487037 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.486931 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv" Sep 10 15:50:00.487037 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.486941 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="563a471dd50b31529d133718410402d9dd70f7af9696d94b77c5f6e8dfc7c16c" Sep 10 15:50:00.596900 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.596874 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:50:00.624746 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.624721 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919956cb-9dfc-47f1-99ce-26222cd8c24a-istiod-ca-cert\") pod \"919956cb-9dfc-47f1-99ce-26222cd8c24a\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " Sep 10 15:50:00.625087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.624767 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-data\") pod \"919956cb-9dfc-47f1-99ce-26222cd8c24a\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " Sep 10 15:50:00.625087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.624894 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-certs\") pod \"919956cb-9dfc-47f1-99ce-26222cd8c24a\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " Sep 10 15:50:00.625087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.624997 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-podinfo\") pod \"919956cb-9dfc-47f1-99ce-26222cd8c24a\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " Sep 10 15:50:00.625087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625024 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-data" (OuterVolumeSpecName: "istio-data") pod "919956cb-9dfc-47f1-99ce-26222cd8c24a" (UID: "919956cb-9dfc-47f1-99ce-26222cd8c24a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:00.625087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625038 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78kvl\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-kube-api-access-78kvl\") pod \"919956cb-9dfc-47f1-99ce-26222cd8c24a\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " Sep 10 15:50:00.625087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625065 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-credential-socket\") pod \"919956cb-9dfc-47f1-99ce-26222cd8c24a\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " Sep 10 15:50:00.625406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625111 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-envoy\") pod \"919956cb-9dfc-47f1-99ce-26222cd8c24a\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " Sep 10 15:50:00.625406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625116 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "919956cb-9dfc-47f1-99ce-26222cd8c24a" (UID: "919956cb-9dfc-47f1-99ce-26222cd8c24a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:00.625406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625136 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919956cb-9dfc-47f1-99ce-26222cd8c24a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "919956cb-9dfc-47f1-99ce-26222cd8c24a" (UID: "919956cb-9dfc-47f1-99ce-26222cd8c24a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:50:00.625406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625148 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-socket\") pod \"919956cb-9dfc-47f1-99ce-26222cd8c24a\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " Sep 10 15:50:00.625406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625192 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-token\") pod \"919956cb-9dfc-47f1-99ce-26222cd8c24a\" (UID: \"919956cb-9dfc-47f1-99ce-26222cd8c24a\") " Sep 10 15:50:00.625406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625331 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "919956cb-9dfc-47f1-99ce-26222cd8c24a" (UID: "919956cb-9dfc-47f1-99ce-26222cd8c24a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:00.625763 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625440 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "919956cb-9dfc-47f1-99ce-26222cd8c24a" (UID: "919956cb-9dfc-47f1-99ce-26222cd8c24a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:00.625763 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625650 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:00.625763 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625669 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:00.625763 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625683 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919956cb-9dfc-47f1-99ce-26222cd8c24a-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:00.625763 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625696 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:00.625763 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.625707 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:00.627490 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.627462 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "919956cb-9dfc-47f1-99ce-26222cd8c24a" (UID: "919956cb-9dfc-47f1-99ce-26222cd8c24a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:00.627490 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.627477 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-kube-api-access-78kvl" (OuterVolumeSpecName: "kube-api-access-78kvl") pod "919956cb-9dfc-47f1-99ce-26222cd8c24a" (UID: "919956cb-9dfc-47f1-99ce-26222cd8c24a"). InnerVolumeSpecName "kube-api-access-78kvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:50:00.627664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.627591 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "919956cb-9dfc-47f1-99ce-26222cd8c24a" (UID: "919956cb-9dfc-47f1-99ce-26222cd8c24a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:50:00.627664 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.627640 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-token" (OuterVolumeSpecName: "istio-token") pod "919956cb-9dfc-47f1-99ce-26222cd8c24a" (UID: "919956cb-9dfc-47f1-99ce-26222cd8c24a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:50:00.726988 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.726956 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:00.726988 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.726983 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:00.726988 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.726993 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919956cb-9dfc-47f1-99ce-26222cd8c24a-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:00.727200 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:00.727001 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78kvl\" (UniqueName: \"kubernetes.io/projected/919956cb-9dfc-47f1-99ce-26222cd8c24a-kube-api-access-78kvl\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:01.125324 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:01.125284 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl"] Sep 10 15:50:01.128263 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:01.128237 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25381b71-t6gfl"] Sep 10 15:50:01.494066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:01.493977 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" event={"ID":"919956cb-9dfc-47f1-99ce-26222cd8c24a","Type":"ContainerDied","Data":"246301a0b864c3ba62b10d185eefee4c9d817471b21878408624b5e1e5a2b846"} Sep 10 15:50:01.494066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:01.494023 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v" Sep 10 15:50:01.494066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:01.494036 2580 scope.go:117] "RemoveContainer" containerID="5a07a3b42a45f7911b077265c4ac1a4ad8ef311bd44ce01429be5e78d7b9afe8" Sep 10 15:50:01.515766 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:01.515739 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v"] Sep 10 15:50:01.520109 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:01.520086 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-f32d75-openshift-default-856585647d-xhm8v"] Sep 10 15:50:02.028301 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:02.028271 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19170c79-f95a-4608-a1a0-b8e35fdd5627" path="/var/lib/kubelet/pods/19170c79-f95a-4608-a1a0-b8e35fdd5627/volumes" Sep 10 15:50:02.028762 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:02.028747 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919956cb-9dfc-47f1-99ce-26222cd8c24a" path="/var/lib/kubelet/pods/919956cb-9dfc-47f1-99ce-26222cd8c24a/volumes" Sep 10 15:50:07.117851 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:07.117815 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv"] Sep 10 15:50:07.119590 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:07.119567 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80d47c-29nxv"] Sep 10 15:50:08.028433 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:08.028389 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfab07f2-78e0-4073-98f3-b804d672ffa8" path="/var/lib/kubelet/pods/bfab07f2-78e0-4073-98f3-b804d672ffa8/volumes" Sep 10 15:50:13.472927 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.472889 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5"] Sep 10 15:50:13.473376 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.473187 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" containerID="cri-o://af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44" gracePeriod=30 Sep 10 15:50:13.819924 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.819811 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk"] Sep 10 15:50:13.820256 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.820240 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919956cb-9dfc-47f1-99ce-26222cd8c24a" containerName="istio-proxy" Sep 10 15:50:13.820308 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.820258 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="919956cb-9dfc-47f1-99ce-26222cd8c24a" containerName="istio-proxy" Sep 10 15:50:13.820308 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.820276 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfab07f2-78e0-4073-98f3-b804d672ffa8" containerName="revoke-keys" Sep 10 15:50:13.820308 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.820282 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfab07f2-78e0-4073-98f3-b804d672ffa8" containerName="revoke-keys" Sep 10 15:50:13.820415 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.820343 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="919956cb-9dfc-47f1-99ce-26222cd8c24a" containerName="istio-proxy" Sep 10 15:50:13.820415 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.820355 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfab07f2-78e0-4073-98f3-b804d672ffa8" containerName="revoke-keys" Sep 10 15:50:13.824728 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.824712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" Sep 10 15:50:13.827090 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.827069 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:50:13.827203 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.827073 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:50:13.830995 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.830970 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk"] Sep 10 15:50:13.961498 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:13.961461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee2482b-5280-4539-a8a2-fd8d51669383-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a850d-29brk\" (UID: \"eee2482b-5280-4539-a8a2-fd8d51669383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" Sep 10 15:50:14.062941 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:14.062894 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee2482b-5280-4539-a8a2-fd8d51669383-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a850d-29brk\" (UID: \"eee2482b-5280-4539-a8a2-fd8d51669383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" Sep 10 15:50:14.063765 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:14.063734 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee2482b-5280-4539-a8a2-fd8d51669383-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a850d-29brk\" (UID: \"eee2482b-5280-4539-a8a2-fd8d51669383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" Sep 10 15:50:14.137728 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:14.137696 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" Sep 10 15:50:14.412268 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:14.412244 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk"] Sep 10 15:50:14.414277 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:50:14.414248 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee2482b_5280_4539_a8a2_fd8d51669383.slice/crio-1f9aa29648272ee2f6d568134d620edacb7e0c7b18c729c63ffd0f475c0a7ad4 WatchSource:0}: Error finding container 1f9aa29648272ee2f6d568134d620edacb7e0c7b18c729c63ffd0f475c0a7ad4: Status 404 returned error can't find the container with id 1f9aa29648272ee2f6d568134d620edacb7e0c7b18c729c63ffd0f475c0a7ad4 Sep 10 15:50:14.563220 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:14.563185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" event={"ID":"eee2482b-5280-4539-a8a2-fd8d51669383","Type":"ContainerStarted","Data":"1f9aa29648272ee2f6d568134d620edacb7e0c7b18c729c63ffd0f475c0a7ad4"} Sep 10 15:50:15.570986 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:15.570890 2580 generic.go:356] "Generic (PLEG): container finished" podID="eee2482b-5280-4539-a8a2-fd8d51669383" containerID="556c8cf1c85f8f8e38b1fe1bb7edc07886789f656001f7af596bec57092afe9f" exitCode=0 Sep 10 15:50:15.570986 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:15.570973 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" event={"ID":"eee2482b-5280-4539-a8a2-fd8d51669383","Type":"ContainerDied","Data":"556c8cf1c85f8f8e38b1fe1bb7edc07886789f656001f7af596bec57092afe9f"} Sep 10 15:50:16.711173 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:16.711150 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" Sep 10 15:50:16.788703 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:16.788666 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee2482b-5280-4539-a8a2-fd8d51669383-maas-api-service-ca\") pod \"eee2482b-5280-4539-a8a2-fd8d51669383\" (UID: \"eee2482b-5280-4539-a8a2-fd8d51669383\") " Sep 10 15:50:16.789036 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:16.789009 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee2482b-5280-4539-a8a2-fd8d51669383-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eee2482b-5280-4539-a8a2-fd8d51669383" (UID: "eee2482b-5280-4539-a8a2-fd8d51669383"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:50:16.890256 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:16.890225 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee2482b-5280-4539-a8a2-fd8d51669383-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:17.583756 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:17.583727 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" Sep 10 15:50:17.583927 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:17.583775 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk" event={"ID":"eee2482b-5280-4539-a8a2-fd8d51669383","Type":"ContainerDied","Data":"1f9aa29648272ee2f6d568134d620edacb7e0c7b18c729c63ffd0f475c0a7ad4"} Sep 10 15:50:17.583927 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:17.583830 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f9aa29648272ee2f6d568134d620edacb7e0c7b18c729c63ffd0f475c0a7ad4" Sep 10 15:50:18.733148 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.733126 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:50:18.811212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811130 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrr8h\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-kube-api-access-mrr8h\") pod \"d0e5a659-ed15-4f90-8b16-076deddba214\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " Sep 10 15:50:18.811212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811161 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e5a659-ed15-4f90-8b16-076deddba214-istiod-ca-cert\") pod \"d0e5a659-ed15-4f90-8b16-076deddba214\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " Sep 10 15:50:18.811212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811197 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-socket\") pod \"d0e5a659-ed15-4f90-8b16-076deddba214\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " Sep 10 15:50:18.811451 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811296 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-envoy\") pod \"d0e5a659-ed15-4f90-8b16-076deddba214\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " Sep 10 15:50:18.811451 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811382 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-credential-socket\") pod \"d0e5a659-ed15-4f90-8b16-076deddba214\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " Sep 10 15:50:18.811574 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811459 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-certs\") pod \"d0e5a659-ed15-4f90-8b16-076deddba214\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " Sep 10 15:50:18.811574 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811508 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d0e5a659-ed15-4f90-8b16-076deddba214" (UID: "d0e5a659-ed15-4f90-8b16-076deddba214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.811574 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811525 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e5a659-ed15-4f90-8b16-076deddba214-istio-podinfo\") pod \"d0e5a659-ed15-4f90-8b16-076deddba214\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " Sep 10 15:50:18.811574 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811564 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-data\") pod \"d0e5a659-ed15-4f90-8b16-076deddba214\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " Sep 10 15:50:18.811776 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811600 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e5a659-ed15-4f90-8b16-076deddba214-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d0e5a659-ed15-4f90-8b16-076deddba214" (UID: "d0e5a659-ed15-4f90-8b16-076deddba214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:50:18.811776 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811642 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-istio-token\") pod \"d0e5a659-ed15-4f90-8b16-076deddba214\" (UID: \"d0e5a659-ed15-4f90-8b16-076deddba214\") " Sep 10 15:50:18.811776 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811699 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d0e5a659-ed15-4f90-8b16-076deddba214" (UID: "d0e5a659-ed15-4f90-8b16-076deddba214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.811902 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811750 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d0e5a659-ed15-4f90-8b16-076deddba214" (UID: "d0e5a659-ed15-4f90-8b16-076deddba214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.811902 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.811814 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-data" (OuterVolumeSpecName: "istio-data") pod "d0e5a659-ed15-4f90-8b16-076deddba214" (UID: "d0e5a659-ed15-4f90-8b16-076deddba214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.812092 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.812070 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.812168 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.812095 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e5a659-ed15-4f90-8b16-076deddba214-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.812168 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.812105 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.812168 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.812113 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.812168 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.812121 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.813592 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.813558 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d0e5a659-ed15-4f90-8b16-076deddba214" (UID: "d0e5a659-ed15-4f90-8b16-076deddba214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:18.813592 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.813583 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-istio-token" (OuterVolumeSpecName: "istio-token") pod "d0e5a659-ed15-4f90-8b16-076deddba214" (UID: "d0e5a659-ed15-4f90-8b16-076deddba214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:50:18.813789 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.813773 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-kube-api-access-mrr8h" (OuterVolumeSpecName: "kube-api-access-mrr8h") pod "d0e5a659-ed15-4f90-8b16-076deddba214" (UID: "d0e5a659-ed15-4f90-8b16-076deddba214"). InnerVolumeSpecName "kube-api-access-mrr8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:50:18.813926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.813908 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0e5a659-ed15-4f90-8b16-076deddba214-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d0e5a659-ed15-4f90-8b16-076deddba214" (UID: "d0e5a659-ed15-4f90-8b16-076deddba214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:50:18.912764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.912732 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.912764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.912758 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrr8h\" (UniqueName: \"kubernetes.io/projected/d0e5a659-ed15-4f90-8b16-076deddba214-kube-api-access-mrr8h\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.912764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.912768 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e5a659-ed15-4f90-8b16-076deddba214-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:18.912978 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:18.912777 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e5a659-ed15-4f90-8b16-076deddba214-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:19.598694 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.598661 2580 generic.go:356] "Generic (PLEG): container finished" podID="d0e5a659-ed15-4f90-8b16-076deddba214" containerID="af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44" exitCode=0 Sep 10 15:50:19.598861 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.598706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" event={"ID":"d0e5a659-ed15-4f90-8b16-076deddba214","Type":"ContainerDied","Data":"af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44"} Sep 10 15:50:19.598861 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.598747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" event={"ID":"d0e5a659-ed15-4f90-8b16-076deddba214","Type":"ContainerDied","Data":"4bfde36df8c9e1cf66497aeb9c08c8ff40a1b8d4ac6d4446ee1a1c00c7a55238"} Sep 10 15:50:19.598861 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.598768 2580 scope.go:117] "RemoveContainer" containerID="af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44" Sep 10 15:50:19.598861 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.598712 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5" Sep 10 15:50:19.608082 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.608067 2580 scope.go:117] "RemoveContainer" containerID="162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7" Sep 10 15:50:19.615946 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.615932 2580 scope.go:117] "RemoveContainer" containerID="af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44" Sep 10 15:50:19.616224 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:50:19.616206 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44\": container with ID starting with af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44 not found: ID does not exist" containerID="af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44" Sep 10 15:50:19.616264 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.616233 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44"} err="failed to get container status \"af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44\": rpc error: code = NotFound desc = could not find container \"af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44\": container with ID starting with af288d78e70a78b22d21a58469bdabac5de14e45b715ddff50986b765cde8a44 not found: ID does not exist" Sep 10 15:50:19.616264 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.616250 2580 scope.go:117] "RemoveContainer" containerID="162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7" Sep 10 15:50:19.616585 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:50:19.616564 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7\": container with ID starting with 162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7 not found: ID does not exist" containerID="162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7" Sep 10 15:50:19.616628 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.616591 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7"} err="failed to get container status \"162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7\": rpc error: code = NotFound desc = could not find container \"162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7\": container with ID starting with 162d98c44314d929a9615db94a703a71ff5e96f3921975ee1895020d7cf3a4e7 not found: ID does not exist" Sep 10 15:50:19.619982 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.619958 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5"] Sep 10 15:50:19.624900 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:19.624877 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-80d47c-openshift-default-5cb47bf4f4-vvkm5"] Sep 10 15:50:20.029029 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:20.028996 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" path="/var/lib/kubelet/pods/d0e5a659-ed15-4f90-8b16-076deddba214/volumes" Sep 10 15:50:24.320748 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.320713 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx"] Sep 10 15:50:24.321561 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.321524 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" podUID="0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24" containerName="payload-processing" containerID="cri-o://b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b" gracePeriod=30 Sep 10 15:50:24.520076 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.520041 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk"] Sep 10 15:50:24.522565 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.522540 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a850d-29brk"] Sep 10 15:50:24.589518 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.589497 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:50:24.629721 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.629689 2580 generic.go:356] "Generic (PLEG): container finished" podID="0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24" containerID="b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b" exitCode=0 Sep 10 15:50:24.629891 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.629725 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" Sep 10 15:50:24.629891 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.629775 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" event={"ID":"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24","Type":"ContainerDied","Data":"b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b"} Sep 10 15:50:24.629891 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.629819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx" event={"ID":"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24","Type":"ContainerDied","Data":"7b69b2e3bd5731f3ac5791f2b4b1e6910b97567d7c328a174eaebc6b1294c2b0"} Sep 10 15:50:24.629891 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.629845 2580 scope.go:117] "RemoveContainer" containerID="b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b" Sep 10 15:50:24.638806 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.638789 2580 scope.go:117] "RemoveContainer" containerID="b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b" Sep 10 15:50:24.639099 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:50:24.639076 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b\": container with ID starting with b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b not found: ID does not exist" containerID="b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b" Sep 10 15:50:24.639191 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.639107 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b"} err="failed to get container status \"b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b\": rpc error: code = NotFound desc = could not find container \"b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b\": container with ID starting with b65edd3c73937f88dce165d8268ed01f1b90077955d8f03ed34fed53d4f9908b not found: ID does not exist" Sep 10 15:50:24.672731 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.672698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-plugins-config-volume\") pod \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\" (UID: \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\") " Sep 10 15:50:24.672892 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.672817 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx6nw\" (UniqueName: \"kubernetes.io/projected/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-kube-api-access-kx6nw\") pod \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\" (UID: \"0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24\") " Sep 10 15:50:24.673097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.673072 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24" (UID: "0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:50:24.674889 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.674860 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-kube-api-access-kx6nw" (OuterVolumeSpecName: "kube-api-access-kx6nw") pod "0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24" (UID: "0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24"). InnerVolumeSpecName "kube-api-access-kx6nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:50:24.774621 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.774589 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:24.774621 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.774615 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx6nw\" (UniqueName: \"kubernetes.io/projected/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24-kube-api-access-kx6nw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:24.950581 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.950550 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx"] Sep 10 15:50:24.959381 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:24.959351 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a850d-6b488cfbc-dbtjx"] Sep 10 15:50:26.029000 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:26.028965 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24" path="/var/lib/kubelet/pods/0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24/volumes" Sep 10 15:50:26.029358 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:26.029310 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee2482b-5280-4539-a8a2-fd8d51669383" path="/var/lib/kubelet/pods/eee2482b-5280-4539-a8a2-fd8d51669383/volumes" Sep 10 15:50:27.037052 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.037020 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9"] Sep 10 15:50:27.037832 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.037808 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24" containerName="payload-processing" Sep 10 15:50:27.037928 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.037840 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24" containerName="payload-processing" Sep 10 15:50:27.037928 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.037871 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" Sep 10 15:50:27.037928 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.037884 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" Sep 10 15:50:27.038076 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.037940 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee2482b-5280-4539-a8a2-fd8d51669383" containerName="revoke-keys" Sep 10 15:50:27.038076 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.037954 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee2482b-5280-4539-a8a2-fd8d51669383" containerName="revoke-keys" Sep 10 15:50:27.038179 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.038085 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f28eb6a-aaa4-47b1-a8a1-f1c6774f1e24" containerName="payload-processing" Sep 10 15:50:27.038179 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.038101 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" Sep 10 15:50:27.038179 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.038111 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee2482b-5280-4539-a8a2-fd8d51669383" containerName="revoke-keys" Sep 10 15:50:27.042269 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.042245 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:27.044845 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.044828 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-8e3dc2-dockercfg-vwp5f\"" Sep 10 15:50:27.045144 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.045120 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-8e3dc2\"" Sep 10 15:50:27.050373 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.050351 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9"] Sep 10 15:50:27.200087 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.200052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9\" (UID: \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:27.200288 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.200108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzjx\" (UniqueName: \"kubernetes.io/projected/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-kube-api-access-qjzjx\") pod \"payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9\" (UID: \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:27.301225 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.301135 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzjx\" (UniqueName: \"kubernetes.io/projected/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-kube-api-access-qjzjx\") pod \"payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9\" (UID: \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:27.301385 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.301252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9\" (UID: \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:27.301964 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.301935 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9\" (UID: \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:27.308585 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.308558 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzjx\" (UniqueName: \"kubernetes.io/projected/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-kube-api-access-qjzjx\") pod \"payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9\" (UID: \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:27.355793 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.355767 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:27.629369 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.629328 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9"] Sep 10 15:50:27.634532 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:50:27.634492 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13767c2_09ca_402e_a8ca_3bb6ff810dc2.slice/crio-4a703cd9e1a9e619233796db052ffde2dc9856afb03ad52337eb15c84755ab42 WatchSource:0}: Error finding container 4a703cd9e1a9e619233796db052ffde2dc9856afb03ad52337eb15c84755ab42: Status 404 returned error can't find the container with id 4a703cd9e1a9e619233796db052ffde2dc9856afb03ad52337eb15c84755ab42 Sep 10 15:50:27.648156 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:27.648131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" event={"ID":"e13767c2-09ca-402e-a8ca-3bb6ff810dc2","Type":"ContainerStarted","Data":"4a703cd9e1a9e619233796db052ffde2dc9856afb03ad52337eb15c84755ab42"} Sep 10 15:50:28.655162 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:28.655129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" event={"ID":"e13767c2-09ca-402e-a8ca-3bb6ff810dc2","Type":"ContainerStarted","Data":"bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433"} Sep 10 15:50:28.655574 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:28.655213 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:28.671182 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:28.671140 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" podStartSLOduration=1.671125446 podStartE2EDuration="1.671125446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:28.669951805 +0000 UTC m=+1087.232260995" watchObservedRunningTime="2026-09-10 15:50:28.671125446 +0000 UTC m=+1087.233434635" Sep 10 15:50:31.053412 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:31.053381 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2"] Sep 10 15:50:31.053873 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:31.053606 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" containerID="cri-o://bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d" gracePeriod=30 Sep 10 15:50:36.307392 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.307366 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:50:36.385994 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.385954 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-podinfo\") pod \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " Sep 10 15:50:36.386180 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386000 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istiod-ca-cert\") pod \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " Sep 10 15:50:36.386180 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386058 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-socket\") pod \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " Sep 10 15:50:36.386180 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386108 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-envoy\") pod \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " Sep 10 15:50:36.386180 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386165 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-data\") pod \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " Sep 10 15:50:36.386440 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386218 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-certs\") pod \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " Sep 10 15:50:36.386440 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-token\") pod \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " Sep 10 15:50:36.386440 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386276 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds4rz\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-kube-api-access-ds4rz\") pod \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " Sep 10 15:50:36.386440 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386308 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-credential-socket\") pod \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\" (UID: \"103dddaa-4e90-4d2c-9c93-184aee3eeccd\") " Sep 10 15:50:36.386665 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386358 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "103dddaa-4e90-4d2c-9c93-184aee3eeccd" (UID: "103dddaa-4e90-4d2c-9c93-184aee3eeccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:50:36.386665 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386456 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-data" (OuterVolumeSpecName: "istio-data") pod "103dddaa-4e90-4d2c-9c93-184aee3eeccd" (UID: "103dddaa-4e90-4d2c-9c93-184aee3eeccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:36.386780 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.386681 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "103dddaa-4e90-4d2c-9c93-184aee3eeccd" (UID: "103dddaa-4e90-4d2c-9c93-184aee3eeccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:36.387314 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.387073 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:36.387314 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.387097 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:36.387314 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.387110 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:36.387314 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.387076 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "103dddaa-4e90-4d2c-9c93-184aee3eeccd" (UID: "103dddaa-4e90-4d2c-9c93-184aee3eeccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:36.387800 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.387771 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "103dddaa-4e90-4d2c-9c93-184aee3eeccd" (UID: "103dddaa-4e90-4d2c-9c93-184aee3eeccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:36.388775 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.388736 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "103dddaa-4e90-4d2c-9c93-184aee3eeccd" (UID: "103dddaa-4e90-4d2c-9c93-184aee3eeccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:50:36.388939 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.388921 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "103dddaa-4e90-4d2c-9c93-184aee3eeccd" (UID: "103dddaa-4e90-4d2c-9c93-184aee3eeccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:50:36.388999 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.388963 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-token" (OuterVolumeSpecName: "istio-token") pod "103dddaa-4e90-4d2c-9c93-184aee3eeccd" (UID: "103dddaa-4e90-4d2c-9c93-184aee3eeccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:50:36.389868 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.389846 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-kube-api-access-ds4rz" (OuterVolumeSpecName: "kube-api-access-ds4rz") pod "103dddaa-4e90-4d2c-9c93-184aee3eeccd" (UID: "103dddaa-4e90-4d2c-9c93-184aee3eeccd"). InnerVolumeSpecName "kube-api-access-ds4rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:50:36.488437 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.488394 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:36.488437 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.488438 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:36.488627 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.488448 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds4rz\" (UniqueName: \"kubernetes.io/projected/103dddaa-4e90-4d2c-9c93-184aee3eeccd-kube-api-access-ds4rz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:36.488627 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.488458 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:36.488627 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.488466 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/103dddaa-4e90-4d2c-9c93-184aee3eeccd-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:36.488627 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.488476 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/103dddaa-4e90-4d2c-9c93-184aee3eeccd-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:50:36.705982 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.705892 2580 generic.go:356] "Generic (PLEG): container finished" podID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerID="bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d" exitCode=0 Sep 10 15:50:36.705982 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.705945 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" Sep 10 15:50:36.706216 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.705983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" event={"ID":"103dddaa-4e90-4d2c-9c93-184aee3eeccd","Type":"ContainerDied","Data":"bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d"} Sep 10 15:50:36.706216 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.706056 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2" event={"ID":"103dddaa-4e90-4d2c-9c93-184aee3eeccd","Type":"ContainerDied","Data":"d5e3f7554e61cb2624a8119ef21b50fc58653557b45003e9b60edd3515a05baa"} Sep 10 15:50:36.706216 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.706088 2580 scope.go:117] "RemoveContainer" containerID="bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d" Sep 10 15:50:36.715476 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.715451 2580 scope.go:117] "RemoveContainer" containerID="bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae" Sep 10 15:50:36.723810 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.723795 2580 scope.go:117] "RemoveContainer" containerID="bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d" Sep 10 15:50:36.724106 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:50:36.724074 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d\": container with ID starting with bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d not found: ID does not exist" containerID="bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d" Sep 10 15:50:36.724145 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.724114 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d"} err="failed to get container status \"bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d\": rpc error: code = NotFound desc = could not find container \"bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d\": container with ID starting with bbfa079604d55bffba7cd178384a7933f1089c8676cb03d723480f88aa64b60d not found: ID does not exist" Sep 10 15:50:36.724145 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.724132 2580 scope.go:117] "RemoveContainer" containerID="bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae" Sep 10 15:50:36.724395 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:50:36.724379 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae\": container with ID starting with bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae not found: ID does not exist" containerID="bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae" Sep 10 15:50:36.724469 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.724403 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae"} err="failed to get container status \"bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae\": rpc error: code = NotFound desc = could not find container \"bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae\": container with ID starting with bc510e17733ef1d425a45c2b2c578bc269f9bc27304824fa7652500ea6c11fae not found: ID does not exist" Sep 10 15:50:36.732168 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.732147 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2"] Sep 10 15:50:36.735839 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.735820 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8a850d-openshift-default-5895888b68-x58h2"] Sep 10 15:50:36.967877 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.967789 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j"] Sep 10 15:50:36.968295 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.968277 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" Sep 10 15:50:36.968390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.968296 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" Sep 10 15:50:36.968390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.968312 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" Sep 10 15:50:36.968390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.968321 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" Sep 10 15:50:36.968390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.968342 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" Sep 10 15:50:36.968390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.968351 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" Sep 10 15:50:36.968730 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.968448 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" Sep 10 15:50:36.968730 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.968463 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e5a659-ed15-4f90-8b16-076deddba214" containerName="istio-proxy" Sep 10 15:50:36.968730 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.968474 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" containerName="istio-proxy" Sep 10 15:50:36.972996 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.972977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:36.975278 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.975260 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-6f8c71\"" Sep 10 15:50:36.975359 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.975299 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-6f8c71-dockercfg-gt8bp\"" Sep 10 15:50:36.980159 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.980138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j"] Sep 10 15:50:36.997097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.993261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e6016f-892e-466a-9e49-8afce7047256-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j\" (UID: \"15e6016f-892e-466a-9e49-8afce7047256\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:36.997097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:36.993349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnp7\" (UniqueName: \"kubernetes.io/projected/15e6016f-892e-466a-9e49-8afce7047256-kube-api-access-cnnp7\") pod \"payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j\" (UID: \"15e6016f-892e-466a-9e49-8afce7047256\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:37.094321 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.094275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnp7\" (UniqueName: \"kubernetes.io/projected/15e6016f-892e-466a-9e49-8afce7047256-kube-api-access-cnnp7\") pod \"payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j\" (UID: \"15e6016f-892e-466a-9e49-8afce7047256\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:37.094517 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.094455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e6016f-892e-466a-9e49-8afce7047256-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j\" (UID: \"15e6016f-892e-466a-9e49-8afce7047256\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:37.095269 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.095246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e6016f-892e-466a-9e49-8afce7047256-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j\" (UID: \"15e6016f-892e-466a-9e49-8afce7047256\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:37.104191 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.104159 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnp7\" (UniqueName: \"kubernetes.io/projected/15e6016f-892e-466a-9e49-8afce7047256-kube-api-access-cnnp7\") pod \"payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j\" (UID: \"15e6016f-892e-466a-9e49-8afce7047256\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:37.286639 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.286564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:37.566196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.566166 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j"] Sep 10 15:50:37.568574 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:50:37.568548 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e6016f_892e_466a_9e49_8afce7047256.slice/crio-2eec615f4870dcef210eec4a15d79d377171ff7c8d9fd68ce5de3c76157c38a0 WatchSource:0}: Error finding container 2eec615f4870dcef210eec4a15d79d377171ff7c8d9fd68ce5de3c76157c38a0: Status 404 returned error can't find the container with id 2eec615f4870dcef210eec4a15d79d377171ff7c8d9fd68ce5de3c76157c38a0 Sep 10 15:50:37.713797 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.713764 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" event={"ID":"15e6016f-892e-466a-9e49-8afce7047256","Type":"ContainerStarted","Data":"6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee"} Sep 10 15:50:37.713989 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.713820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" event={"ID":"15e6016f-892e-466a-9e49-8afce7047256","Type":"ContainerStarted","Data":"2eec615f4870dcef210eec4a15d79d377171ff7c8d9fd68ce5de3c76157c38a0"} Sep 10 15:50:37.713989 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.713853 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:37.729970 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:37.729898 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" podStartSLOduration=1.729882629 podStartE2EDuration="1.729882629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:37.728856002 +0000 UTC m=+1096.291165196" watchObservedRunningTime="2026-09-10 15:50:37.729882629 +0000 UTC m=+1096.292191819" Sep 10 15:50:38.032029 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:38.031997 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103dddaa-4e90-4d2c-9c93-184aee3eeccd" path="/var/lib/kubelet/pods/103dddaa-4e90-4d2c-9c93-184aee3eeccd/volumes" Sep 10 15:50:39.663586 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:39.663555 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:50:48.721745 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:48.721711 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 15:50:55.170908 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.170871 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94"] Sep 10 15:50:55.174300 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.174276 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:50:55.176719 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.176696 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-761990\"" Sep 10 15:50:55.176814 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.176754 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-761990-dockercfg-ph4kz\"" Sep 10 15:50:55.184993 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.184969 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94"] Sep 10 15:50:55.267416 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.267377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgjw8\" (UniqueName: \"kubernetes.io/projected/562e010c-8db8-4a06-89c1-cfc6de11277b-kube-api-access-tgjw8\") pod \"payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94\" (UID: \"562e010c-8db8-4a06-89c1-cfc6de11277b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:50:55.267590 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.267490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562e010c-8db8-4a06-89c1-cfc6de11277b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94\" (UID: \"562e010c-8db8-4a06-89c1-cfc6de11277b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:50:55.304926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.304886 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx"] Sep 10 15:50:55.308842 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.308818 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:50:55.318222 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.318192 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx"] Sep 10 15:50:55.368564 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.368532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgjw8\" (UniqueName: \"kubernetes.io/projected/562e010c-8db8-4a06-89c1-cfc6de11277b-kube-api-access-tgjw8\") pod \"payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94\" (UID: \"562e010c-8db8-4a06-89c1-cfc6de11277b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:50:55.368761 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.368591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562e010c-8db8-4a06-89c1-cfc6de11277b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94\" (UID: \"562e010c-8db8-4a06-89c1-cfc6de11277b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:50:55.369284 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.369262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562e010c-8db8-4a06-89c1-cfc6de11277b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94\" (UID: \"562e010c-8db8-4a06-89c1-cfc6de11277b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:50:55.380006 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.379979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgjw8\" (UniqueName: \"kubernetes.io/projected/562e010c-8db8-4a06-89c1-cfc6de11277b-kube-api-access-tgjw8\") pod \"payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94\" (UID: \"562e010c-8db8-4a06-89c1-cfc6de11277b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:50:55.469175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.469077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142424f-1159-4934-9f39-c59cebcfb54c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx\" (UID: \"9142424f-1159-4934-9f39-c59cebcfb54c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:50:55.469175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.469117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqqjw\" (UniqueName: \"kubernetes.io/projected/9142424f-1159-4934-9f39-c59cebcfb54c-kube-api-access-zqqjw\") pod \"payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx\" (UID: \"9142424f-1159-4934-9f39-c59cebcfb54c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:50:55.484963 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.484933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:50:55.570233 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.569887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142424f-1159-4934-9f39-c59cebcfb54c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx\" (UID: \"9142424f-1159-4934-9f39-c59cebcfb54c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:50:55.570672 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.570649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqqjw\" (UniqueName: \"kubernetes.io/projected/9142424f-1159-4934-9f39-c59cebcfb54c-kube-api-access-zqqjw\") pod \"payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx\" (UID: \"9142424f-1159-4934-9f39-c59cebcfb54c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:50:55.570829 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.570783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142424f-1159-4934-9f39-c59cebcfb54c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx\" (UID: \"9142424f-1159-4934-9f39-c59cebcfb54c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:50:55.578968 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.578930 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqqjw\" (UniqueName: \"kubernetes.io/projected/9142424f-1159-4934-9f39-c59cebcfb54c-kube-api-access-zqqjw\") pod \"payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx\" (UID: \"9142424f-1159-4934-9f39-c59cebcfb54c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:50:55.621134 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.621103 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:50:55.759710 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.759686 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94"] Sep 10 15:50:55.760786 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:50:55.760761 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562e010c_8db8_4a06_89c1_cfc6de11277b.slice/crio-b54a2035064f362028e42f1026e9c511306c641d9dca7b811e3a9c6fc088b852 WatchSource:0}: Error finding container b54a2035064f362028e42f1026e9c511306c641d9dca7b811e3a9c6fc088b852: Status 404 returned error can't find the container with id b54a2035064f362028e42f1026e9c511306c641d9dca7b811e3a9c6fc088b852 Sep 10 15:50:55.820737 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.820703 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" event={"ID":"562e010c-8db8-4a06-89c1-cfc6de11277b","Type":"ContainerStarted","Data":"b54a2035064f362028e42f1026e9c511306c641d9dca7b811e3a9c6fc088b852"} Sep 10 15:50:55.893605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:55.893577 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx"] Sep 10 15:50:55.895382 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:50:55.895358 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9142424f_1159_4934_9f39_c59cebcfb54c.slice/crio-a08bfefc7b6abde79952c55e20035a143f8293477f5669533c672b85994b6b40 WatchSource:0}: Error finding container a08bfefc7b6abde79952c55e20035a143f8293477f5669533c672b85994b6b40: Status 404 returned error can't find the container with id a08bfefc7b6abde79952c55e20035a143f8293477f5669533c672b85994b6b40 Sep 10 15:50:56.828004 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:56.827969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" event={"ID":"9142424f-1159-4934-9f39-c59cebcfb54c","Type":"ContainerStarted","Data":"26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97"} Sep 10 15:50:56.828478 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:56.828027 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" event={"ID":"9142424f-1159-4934-9f39-c59cebcfb54c","Type":"ContainerStarted","Data":"a08bfefc7b6abde79952c55e20035a143f8293477f5669533c672b85994b6b40"} Sep 10 15:50:56.828478 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:56.828056 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:50:56.829782 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:56.829741 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" event={"ID":"562e010c-8db8-4a06-89c1-cfc6de11277b","Type":"ContainerStarted","Data":"9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910"} Sep 10 15:50:56.829919 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:56.829898 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:50:56.847238 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:56.847190 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" podStartSLOduration=1.8471755920000001 podStartE2EDuration="1.847175592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:56.844934491 +0000 UTC m=+1115.407243684" watchObservedRunningTime="2026-09-10 15:50:56.847175592 +0000 UTC m=+1115.409484773" Sep 10 15:50:56.862907 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:50:56.862863 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" podStartSLOduration=1.8628490229999999 podStartE2EDuration="1.862849023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:56.860323267 +0000 UTC m=+1115.422632459" watchObservedRunningTime="2026-09-10 15:50:56.862849023 +0000 UTC m=+1115.425158212" Sep 10 15:51:05.065973 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.065933 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk"] Sep 10 15:51:05.070846 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.070824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:05.073175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.073158 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e412c4\"" Sep 10 15:51:05.073278 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.073225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e412c4-dockercfg-xg8hg\"" Sep 10 15:51:05.081127 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.081107 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk"] Sep 10 15:51:05.098925 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.098890 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw"] Sep 10 15:51:05.103370 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.103346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:05.111844 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.111715 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw"] Sep 10 15:51:05.269949 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.269896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhw5x\" (UniqueName: \"kubernetes.io/projected/bb38e674-04ab-4baf-8df9-f84f91af92f6-kube-api-access-jhw5x\") pod \"payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw\" (UID: \"bb38e674-04ab-4baf-8df9-f84f91af92f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:05.269949 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.269947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvtq\" (UniqueName: \"kubernetes.io/projected/b76436ed-9465-40a1-8e35-c3664cea02f2-kube-api-access-kzvtq\") pod \"payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk\" (UID: \"b76436ed-9465-40a1-8e35-c3664cea02f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:05.270215 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.270093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb38e674-04ab-4baf-8df9-f84f91af92f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw\" (UID: \"bb38e674-04ab-4baf-8df9-f84f91af92f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:05.270215 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.270200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76436ed-9465-40a1-8e35-c3664cea02f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk\" (UID: \"b76436ed-9465-40a1-8e35-c3664cea02f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:05.370984 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.370952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76436ed-9465-40a1-8e35-c3664cea02f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk\" (UID: \"b76436ed-9465-40a1-8e35-c3664cea02f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:05.371144 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.371020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhw5x\" (UniqueName: \"kubernetes.io/projected/bb38e674-04ab-4baf-8df9-f84f91af92f6-kube-api-access-jhw5x\") pod \"payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw\" (UID: \"bb38e674-04ab-4baf-8df9-f84f91af92f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:05.371144 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.371043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvtq\" (UniqueName: \"kubernetes.io/projected/b76436ed-9465-40a1-8e35-c3664cea02f2-kube-api-access-kzvtq\") pod \"payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk\" (UID: \"b76436ed-9465-40a1-8e35-c3664cea02f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:05.371144 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.371124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb38e674-04ab-4baf-8df9-f84f91af92f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw\" (UID: \"bb38e674-04ab-4baf-8df9-f84f91af92f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:05.371682 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.371663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76436ed-9465-40a1-8e35-c3664cea02f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk\" (UID: \"b76436ed-9465-40a1-8e35-c3664cea02f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:05.371791 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.371769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb38e674-04ab-4baf-8df9-f84f91af92f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw\" (UID: \"bb38e674-04ab-4baf-8df9-f84f91af92f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:05.378795 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.378769 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvtq\" (UniqueName: \"kubernetes.io/projected/b76436ed-9465-40a1-8e35-c3664cea02f2-kube-api-access-kzvtq\") pod \"payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk\" (UID: \"b76436ed-9465-40a1-8e35-c3664cea02f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:05.378795 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.378785 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhw5x\" (UniqueName: \"kubernetes.io/projected/bb38e674-04ab-4baf-8df9-f84f91af92f6-kube-api-access-jhw5x\") pod \"payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw\" (UID: \"bb38e674-04ab-4baf-8df9-f84f91af92f6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:05.383600 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.383578 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:05.416043 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.416011 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:05.657818 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.657790 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk"] Sep 10 15:51:05.659904 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:51:05.659874 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76436ed_9465_40a1_8e35_c3664cea02f2.slice/crio-5163a2fe6e4c79115321263cc9d59179e38a1f62a262cbba3761666bafc4e9c3 WatchSource:0}: Error finding container 5163a2fe6e4c79115321263cc9d59179e38a1f62a262cbba3761666bafc4e9c3: Status 404 returned error can't find the container with id 5163a2fe6e4c79115321263cc9d59179e38a1f62a262cbba3761666bafc4e9c3 Sep 10 15:51:05.702340 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.702317 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw"] Sep 10 15:51:05.705614 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:51:05.705587 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb38e674_04ab_4baf_8df9_f84f91af92f6.slice/crio-7d727819071b34afd1a782edf8e14a27585df6f9f879f4acfdfeb97f0ee2f9a3 WatchSource:0}: Error finding container 7d727819071b34afd1a782edf8e14a27585df6f9f879f4acfdfeb97f0ee2f9a3: Status 404 returned error can't find the container with id 7d727819071b34afd1a782edf8e14a27585df6f9f879f4acfdfeb97f0ee2f9a3 Sep 10 15:51:05.885286 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.885250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" event={"ID":"b76436ed-9465-40a1-8e35-c3664cea02f2","Type":"ContainerStarted","Data":"1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284"} Sep 10 15:51:05.885508 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.885321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" event={"ID":"b76436ed-9465-40a1-8e35-c3664cea02f2","Type":"ContainerStarted","Data":"5163a2fe6e4c79115321263cc9d59179e38a1f62a262cbba3761666bafc4e9c3"} Sep 10 15:51:05.885508 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.885351 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:05.886982 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.886955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" event={"ID":"bb38e674-04ab-4baf-8df9-f84f91af92f6","Type":"ContainerStarted","Data":"d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c"} Sep 10 15:51:05.887107 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.886996 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" event={"ID":"bb38e674-04ab-4baf-8df9-f84f91af92f6","Type":"ContainerStarted","Data":"7d727819071b34afd1a782edf8e14a27585df6f9f879f4acfdfeb97f0ee2f9a3"} Sep 10 15:51:05.887107 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.887016 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:05.902925 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.902874 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" podStartSLOduration=0.90286149 podStartE2EDuration="902.86149ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:05.901862671 +0000 UTC m=+1124.464171861" watchObservedRunningTime="2026-09-10 15:51:05.90286149 +0000 UTC m=+1124.465170680" Sep 10 15:51:05.920280 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:05.920182 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" podStartSLOduration=0.920159571 podStartE2EDuration="920.159571ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:05.916617868 +0000 UTC m=+1124.478927062" watchObservedRunningTime="2026-09-10 15:51:05.920159571 +0000 UTC m=+1124.482468762" Sep 10 15:51:07.837867 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:07.837837 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:51:07.838280 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:07.837907 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:51:16.895117 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:16.895081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 15:51:16.895537 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:16.895157 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 15:51:46.518163 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.518091 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r"] Sep 10 15:51:46.523406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.523376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:51:46.525804 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.525781 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c1384d3d\"" Sep 10 15:51:46.525924 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.525843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c1384d3d-dockercfg-md52k\"" Sep 10 15:51:46.532922 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.532894 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r"] Sep 10 15:51:46.553537 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.553517 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r"] Sep 10 15:51:46.561396 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.561366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptrvv\" (UniqueName: \"kubernetes.io/projected/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-kube-api-access-ptrvv\") pod \"payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r\" (UID: \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\") " pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:51:46.561645 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.561624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r\" (UID: \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\") " pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:51:46.563172 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.563148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:46.567963 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.567939 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r"] Sep 10 15:51:46.663166 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.663134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptrvv\" (UniqueName: \"kubernetes.io/projected/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-kube-api-access-ptrvv\") pod \"payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r\" (UID: \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\") " pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:51:46.663351 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.663191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r\" (UID: \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:46.663351 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.663237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r\" (UID: \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\") " pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:51:46.663351 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.663281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dhx\" (UniqueName: \"kubernetes.io/projected/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-kube-api-access-f8dhx\") pod \"payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r\" (UID: \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:46.664001 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.663975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r\" (UID: \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\") " pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:51:46.671167 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.671142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptrvv\" (UniqueName: \"kubernetes.io/projected/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-kube-api-access-ptrvv\") pod \"payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r\" (UID: \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\") " pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:51:46.764119 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.764075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r\" (UID: \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:46.764316 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.764147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dhx\" (UniqueName: \"kubernetes.io/projected/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-kube-api-access-f8dhx\") pod \"payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r\" (UID: \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:46.764807 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.764786 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r\" (UID: \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:46.772325 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.772263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dhx\" (UniqueName: \"kubernetes.io/projected/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-kube-api-access-f8dhx\") pod \"payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r\" (UID: \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:46.835518 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.835485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:51:46.874895 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:46.874832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:47.113618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:47.113590 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r"] Sep 10 15:51:47.117551 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:51:47.117518 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fbe7a5_24cc_4513_bd7d_12749a4ff4e1.slice/crio-bbc566990cdaa8dec4782fc3fa9d48db5e3c4d74d24d1ccc9253a93394ceab5e WatchSource:0}: Error finding container bbc566990cdaa8dec4782fc3fa9d48db5e3c4d74d24d1ccc9253a93394ceab5e: Status 404 returned error can't find the container with id bbc566990cdaa8dec4782fc3fa9d48db5e3c4d74d24d1ccc9253a93394ceab5e Sep 10 15:51:47.161304 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:47.161260 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r"] Sep 10 15:51:47.171540 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:51:47.171517 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ff89ea_54fa_4e9b_ba97_a3584da2847e.slice/crio-2b719642b86a9fbd73aa21dd51b5507605f5895756df2704305f1459198751fc WatchSource:0}: Error finding container 2b719642b86a9fbd73aa21dd51b5507605f5895756df2704305f1459198751fc: Status 404 returned error can't find the container with id 2b719642b86a9fbd73aa21dd51b5507605f5895756df2704305f1459198751fc Sep 10 15:51:48.125480 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:48.125432 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" event={"ID":"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1","Type":"ContainerStarted","Data":"92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90"} Sep 10 15:51:48.125940 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:48.125495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" event={"ID":"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1","Type":"ContainerStarted","Data":"bbc566990cdaa8dec4782fc3fa9d48db5e3c4d74d24d1ccc9253a93394ceab5e"} Sep 10 15:51:48.125940 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:48.125523 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:51:48.127337 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:48.127310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" event={"ID":"d0ff89ea-54fa-4e9b-ba97-a3584da2847e","Type":"ContainerStarted","Data":"0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641"} Sep 10 15:51:48.127508 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:48.127356 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" event={"ID":"d0ff89ea-54fa-4e9b-ba97-a3584da2847e","Type":"ContainerStarted","Data":"2b719642b86a9fbd73aa21dd51b5507605f5895756df2704305f1459198751fc"} Sep 10 15:51:48.127508 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:48.127387 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:48.142836 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:48.142789 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" podStartSLOduration=2.142774751 podStartE2EDuration="2.142774751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:48.139996462 +0000 UTC m=+1166.702305655" watchObservedRunningTime="2026-09-10 15:51:48.142774751 +0000 UTC m=+1166.705083941" Sep 10 15:51:48.155675 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:48.155633 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" podStartSLOduration=2.155620076 podStartE2EDuration="2.155620076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:48.153756953 +0000 UTC m=+1166.716066144" watchObservedRunningTime="2026-09-10 15:51:48.155620076 +0000 UTC m=+1166.717929266" Sep 10 15:51:59.135610 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:59.135582 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:51:59.136077 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:51:59.135901 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:52:22.036395 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:22.036368 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:52:22.038723 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:22.038703 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:52:22.137306 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:22.137273 2580 scope.go:117] "RemoveContainer" containerID="17844b0d57ef731b5ccfec4c34ff479045ccdc6d7de8428d216f70ce741d4f6d" Sep 10 15:52:22.145934 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:22.145906 2580 scope.go:117] "RemoveContainer" containerID="6954ed3002a2234334c98fdce003d901f272a7f4c48d29f24ccd23ae6f224ed4" Sep 10 15:52:49.236080 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.236036 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn"] Sep 10 15:52:49.239720 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.239699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" Sep 10 15:52:49.242056 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.242033 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:52:49.242173 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.242033 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:52:49.246216 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.246187 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn"] Sep 10 15:52:49.392207 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.392171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ad540-0dbf-4dfc-b089-76d602984cda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn\" (UID: \"d78ad540-0dbf-4dfc-b089-76d602984cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" Sep 10 15:52:49.493371 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.493279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ad540-0dbf-4dfc-b089-76d602984cda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn\" (UID: \"d78ad540-0dbf-4dfc-b089-76d602984cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" Sep 10 15:52:49.493950 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.493922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ad540-0dbf-4dfc-b089-76d602984cda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn\" (UID: \"d78ad540-0dbf-4dfc-b089-76d602984cda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" Sep 10 15:52:49.553144 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.553109 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" Sep 10 15:52:49.830404 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:49.830376 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn"] Sep 10 15:52:49.832539 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:52:49.832505 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78ad540_0dbf_4dfc_b089_76d602984cda.slice/crio-663d116cb9ede1be2bda061c49fba904d5f36835c3a68f068be55702522b5395 WatchSource:0}: Error finding container 663d116cb9ede1be2bda061c49fba904d5f36835c3a68f068be55702522b5395: Status 404 returned error can't find the container with id 663d116cb9ede1be2bda061c49fba904d5f36835c3a68f068be55702522b5395 Sep 10 15:52:50.473489 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:50.473399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" event={"ID":"d78ad540-0dbf-4dfc-b089-76d602984cda","Type":"ContainerStarted","Data":"e1cc6f6d08f666b14cf52353fd2a05f2b9d32768de21a155ca60c09f8342ed84"} Sep 10 15:52:50.473489 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:50.473470 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" event={"ID":"d78ad540-0dbf-4dfc-b089-76d602984cda","Type":"ContainerStarted","Data":"663d116cb9ede1be2bda061c49fba904d5f36835c3a68f068be55702522b5395"} Sep 10 15:52:50.490517 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:50.490469 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" podStartSLOduration=1.490454362 podStartE2EDuration="1.490454362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:50.487805516 +0000 UTC m=+1229.050114707" watchObservedRunningTime="2026-09-10 15:52:50.490454362 +0000 UTC m=+1229.052763766" Sep 10 15:52:51.481109 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:51.481069 2580 generic.go:356] "Generic (PLEG): container finished" podID="d78ad540-0dbf-4dfc-b089-76d602984cda" containerID="e1cc6f6d08f666b14cf52353fd2a05f2b9d32768de21a155ca60c09f8342ed84" exitCode=0 Sep 10 15:52:51.481487 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:51.481151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" event={"ID":"d78ad540-0dbf-4dfc-b089-76d602984cda","Type":"ContainerDied","Data":"e1cc6f6d08f666b14cf52353fd2a05f2b9d32768de21a155ca60c09f8342ed84"} Sep 10 15:52:52.622528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:52.622504 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" Sep 10 15:52:52.725303 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:52.725269 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ad540-0dbf-4dfc-b089-76d602984cda-maas-api-service-ca\") pod \"d78ad540-0dbf-4dfc-b089-76d602984cda\" (UID: \"d78ad540-0dbf-4dfc-b089-76d602984cda\") " Sep 10 15:52:52.725662 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:52.725639 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78ad540-0dbf-4dfc-b089-76d602984cda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d78ad540-0dbf-4dfc-b089-76d602984cda" (UID: "d78ad540-0dbf-4dfc-b089-76d602984cda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:52:52.826614 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:52.826526 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78ad540-0dbf-4dfc-b089-76d602984cda-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:52:53.492548 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:53.492513 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" Sep 10 15:52:53.492548 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:53.492542 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn" event={"ID":"d78ad540-0dbf-4dfc-b089-76d602984cda","Type":"ContainerDied","Data":"663d116cb9ede1be2bda061c49fba904d5f36835c3a68f068be55702522b5395"} Sep 10 15:52:53.492776 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:52:53.492596 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="663d116cb9ede1be2bda061c49fba904d5f36835c3a68f068be55702522b5395" Sep 10 15:53:04.758725 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:04.758685 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r"] Sep 10 15:53:04.759194 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:04.758953 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" podUID="46fbe7a5-24cc-4513-bd7d-12749a4ff4e1" containerName="payload-processing" containerID="cri-o://92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90" gracePeriod=30 Sep 10 15:53:04.765604 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:04.765574 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r"] Sep 10 15:53:04.765818 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:04.765795 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" podUID="d0ff89ea-54fa-4e9b-ba97-a3584da2847e" containerName="payload-pre-processing" containerID="cri-o://0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641" gracePeriod=30 Sep 10 15:53:04.932668 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:04.932629 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn"] Sep 10 15:53:04.934491 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:04.934465 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1384d3d-8xbrn"] Sep 10 15:53:05.053221 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.053194 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:53:05.056024 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.056001 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:53:05.149149 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.149115 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-plugins-config-volume\") pod \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\" (UID: \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\") " Sep 10 15:53:05.149336 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.149163 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8dhx\" (UniqueName: \"kubernetes.io/projected/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-kube-api-access-f8dhx\") pod \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\" (UID: \"d0ff89ea-54fa-4e9b-ba97-a3584da2847e\") " Sep 10 15:53:05.149336 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.149189 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-plugins-config-volume\") pod \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\" (UID: \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\") " Sep 10 15:53:05.149336 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.149254 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptrvv\" (UniqueName: \"kubernetes.io/projected/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-kube-api-access-ptrvv\") pod \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\" (UID: \"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1\") " Sep 10 15:53:05.149553 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.149513 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46fbe7a5-24cc-4513-bd7d-12749a4ff4e1" (UID: "46fbe7a5-24cc-4513-bd7d-12749a4ff4e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:53:05.149553 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.149535 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0ff89ea-54fa-4e9b-ba97-a3584da2847e" (UID: "d0ff89ea-54fa-4e9b-ba97-a3584da2847e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:53:05.149655 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.149631 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:53:05.149655 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.149644 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:53:05.151385 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.151362 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-kube-api-access-f8dhx" (OuterVolumeSpecName: "kube-api-access-f8dhx") pod "d0ff89ea-54fa-4e9b-ba97-a3584da2847e" (UID: "d0ff89ea-54fa-4e9b-ba97-a3584da2847e"). InnerVolumeSpecName "kube-api-access-f8dhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:53:05.151494 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.151385 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-kube-api-access-ptrvv" (OuterVolumeSpecName: "kube-api-access-ptrvv") pod "46fbe7a5-24cc-4513-bd7d-12749a4ff4e1" (UID: "46fbe7a5-24cc-4513-bd7d-12749a4ff4e1"). InnerVolumeSpecName "kube-api-access-ptrvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:53:05.250955 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.250915 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8dhx\" (UniqueName: \"kubernetes.io/projected/d0ff89ea-54fa-4e9b-ba97-a3584da2847e-kube-api-access-f8dhx\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:53:05.250955 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.250947 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptrvv\" (UniqueName: \"kubernetes.io/projected/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1-kube-api-access-ptrvv\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:53:05.563022 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.562980 2580 generic.go:356] "Generic (PLEG): container finished" podID="46fbe7a5-24cc-4513-bd7d-12749a4ff4e1" containerID="92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90" exitCode=0 Sep 10 15:53:05.563022 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.563025 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" Sep 10 15:53:05.563276 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.563064 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" event={"ID":"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1","Type":"ContainerDied","Data":"92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90"} Sep 10 15:53:05.563276 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.563115 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r" event={"ID":"46fbe7a5-24cc-4513-bd7d-12749a4ff4e1","Type":"ContainerDied","Data":"bbc566990cdaa8dec4782fc3fa9d48db5e3c4d74d24d1ccc9253a93394ceab5e"} Sep 10 15:53:05.563276 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.563144 2580 scope.go:117] "RemoveContainer" containerID="92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90" Sep 10 15:53:05.564694 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.564673 2580 generic.go:356] "Generic (PLEG): container finished" podID="d0ff89ea-54fa-4e9b-ba97-a3584da2847e" containerID="0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641" exitCode=0 Sep 10 15:53:05.564824 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.564724 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" Sep 10 15:53:05.564824 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.564731 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" event={"ID":"d0ff89ea-54fa-4e9b-ba97-a3584da2847e","Type":"ContainerDied","Data":"0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641"} Sep 10 15:53:05.564824 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.564769 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r" event={"ID":"d0ff89ea-54fa-4e9b-ba97-a3584da2847e","Type":"ContainerDied","Data":"2b719642b86a9fbd73aa21dd51b5507605f5895756df2704305f1459198751fc"} Sep 10 15:53:05.574391 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.574374 2580 scope.go:117] "RemoveContainer" containerID="92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90" Sep 10 15:53:05.574714 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:53:05.574688 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90\": container with ID starting with 92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90 not found: ID does not exist" containerID="92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90" Sep 10 15:53:05.574812 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.574721 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90"} err="failed to get container status \"92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90\": rpc error: code = NotFound desc = could not find container \"92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90\": container with ID starting with 92807de4993492e7e637e115a927dc4328a6b7ed3ef9042f3055602f10b1ec90 not found: ID does not exist" Sep 10 15:53:05.574812 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.574739 2580 scope.go:117] "RemoveContainer" containerID="0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641" Sep 10 15:53:05.583175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.583155 2580 scope.go:117] "RemoveContainer" containerID="0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641" Sep 10 15:53:05.583497 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:53:05.583470 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641\": container with ID starting with 0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641 not found: ID does not exist" containerID="0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641" Sep 10 15:53:05.583590 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.583506 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641"} err="failed to get container status \"0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641\": rpc error: code = NotFound desc = could not find container \"0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641\": container with ID starting with 0628cd7aff54569ed34e9c0afdfa2fc25dd02717eab2240dd3c99f7bb2fdf641 not found: ID does not exist" Sep 10 15:53:05.585180 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.585162 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r"] Sep 10 15:53:05.592485 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.592460 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c1384d3d-d99758f89-hv75r"] Sep 10 15:53:05.600293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.600272 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r"] Sep 10 15:53:05.606214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:05.606194 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c1384d3d-6fdc4594bb-5rs4r"] Sep 10 15:53:06.033802 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:06.030836 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46fbe7a5-24cc-4513-bd7d-12749a4ff4e1" path="/var/lib/kubelet/pods/46fbe7a5-24cc-4513-bd7d-12749a4ff4e1/volumes" Sep 10 15:53:06.033802 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:06.031445 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0ff89ea-54fa-4e9b-ba97-a3584da2847e" path="/var/lib/kubelet/pods/d0ff89ea-54fa-4e9b-ba97-a3584da2847e/volumes" Sep 10 15:53:06.033802 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:06.031921 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78ad540-0dbf-4dfc-b089-76d602984cda" path="/var/lib/kubelet/pods/d78ad540-0dbf-4dfc-b089-76d602984cda/volumes" Sep 10 15:53:22.173174 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:53:22.173100 2580 scope.go:117] "RemoveContainer" containerID="fb9b5ab295f8e80e8aa4530200e0b1aa981d4cc5247ae876c4b37abe9f11a216" Sep 10 15:54:22.202469 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:22.202416 2580 scope.go:117] "RemoveContainer" containerID="781dbfda1daaa5ac9913df0fff5f314daea62ea8ecd77fd1b6846db8535ca282" Sep 10 15:54:22.212804 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:22.212786 2580 scope.go:117] "RemoveContainer" containerID="49d274446277c5e553a880f9a056c99103ed37204055113b1af367516f56d2cb" Sep 10 15:54:45.206861 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.206778 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8"] Sep 10 15:54:45.207527 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.207507 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78ad540-0dbf-4dfc-b089-76d602984cda" containerName="revoke-keys" Sep 10 15:54:45.207621 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.207531 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78ad540-0dbf-4dfc-b089-76d602984cda" containerName="revoke-keys" Sep 10 15:54:45.207621 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.207548 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46fbe7a5-24cc-4513-bd7d-12749a4ff4e1" containerName="payload-processing" Sep 10 15:54:45.207621 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.207556 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="46fbe7a5-24cc-4513-bd7d-12749a4ff4e1" containerName="payload-processing" Sep 10 15:54:45.207621 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.207565 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ff89ea-54fa-4e9b-ba97-a3584da2847e" containerName="payload-pre-processing" Sep 10 15:54:45.207621 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.207575 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ff89ea-54fa-4e9b-ba97-a3584da2847e" containerName="payload-pre-processing" Sep 10 15:54:45.207871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.207677 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ff89ea-54fa-4e9b-ba97-a3584da2847e" containerName="payload-pre-processing" Sep 10 15:54:45.207871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.207690 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="46fbe7a5-24cc-4513-bd7d-12749a4ff4e1" containerName="payload-processing" Sep 10 15:54:45.207871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.207701 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78ad540-0dbf-4dfc-b089-76d602984cda" containerName="revoke-keys" Sep 10 15:54:45.211409 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.211383 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.213781 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.213762 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a653ab-openshift-default-dockercfg-kdt94\"" Sep 10 15:54:45.226648 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.226617 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8"] Sep 10 15:54:45.386702 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.386667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f068f0-bf96-4710-9a30-a434355e2260-istiod-ca-cert\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.386702 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.386707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-istio-token\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.386936 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.386786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-envoy\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.386936 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.386831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2p5\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-kube-api-access-8l2p5\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.386936 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.386848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-socket\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.386936 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.386883 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-certs\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.387137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.386942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-credential-socket\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.387137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.387014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-data\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.387137 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.387038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f068f0-bf96-4710-9a30-a434355e2260-istio-podinfo\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-data\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488140 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f068f0-bf96-4710-9a30-a434355e2260-istio-podinfo\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488187 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f068f0-bf96-4710-9a30-a434355e2260-istiod-ca-cert\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488470 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-istio-token\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488470 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-envoy\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488470 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2p5\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-kube-api-access-8l2p5\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488639 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488411 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-socket\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488639 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-certs\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488639 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488607 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-credential-socket\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488800 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-socket\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488800 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-data\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.488980 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488958 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-certs\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.489075 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.489004 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-credential-socket\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.489075 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.488984 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f068f0-bf96-4710-9a30-a434355e2260-istiod-ca-cert\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.491109 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.491086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-envoy\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.491379 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.491360 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f068f0-bf96-4710-9a30-a434355e2260-istio-podinfo\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.495657 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.495626 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-istio-token\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.495921 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.495900 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2p5\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-kube-api-access-8l2p5\") pod \"e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.528000 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.527974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:45.804081 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.804051 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8"] Sep 10 15:54:45.806355 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:54:45.806325 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f068f0_bf96_4710_9a30_a434355e2260.slice/crio-438e1b9f0f545f536caa2d0b2ad2e31828fdec5c9b23cc3eee368da382672648 WatchSource:0}: Error finding container 438e1b9f0f545f536caa2d0b2ad2e31828fdec5c9b23cc3eee368da382672648: Status 404 returned error can't find the container with id 438e1b9f0f545f536caa2d0b2ad2e31828fdec5c9b23cc3eee368da382672648 Sep 10 15:54:45.808821 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.808791 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:54:45.808899 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.808853 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:54:45.808899 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:45.808881 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:54:46.102155 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:46.102073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" event={"ID":"72f068f0-bf96-4710-9a30-a434355e2260","Type":"ContainerStarted","Data":"1b7f1d745e699b37121eefa9afdc6948462e93154c7b2a2afaba6dbfd2ba4713"} Sep 10 15:54:46.102155 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:46.102138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" event={"ID":"72f068f0-bf96-4710-9a30-a434355e2260","Type":"ContainerStarted","Data":"438e1b9f0f545f536caa2d0b2ad2e31828fdec5c9b23cc3eee368da382672648"} Sep 10 15:54:46.122043 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:46.121990 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" podStartSLOduration=1.121975572 podStartE2EDuration="1.121975572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:46.120885675 +0000 UTC m=+1344.683194865" watchObservedRunningTime="2026-09-10 15:54:46.121975572 +0000 UTC m=+1344.684284762" Sep 10 15:54:46.528777 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:46.528736 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:46.533956 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:46.533932 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:47.108034 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:47.108004 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:47.109018 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:47.108997 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:54:49.808800 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:49.808765 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7"] Sep 10 15:54:49.812731 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:49.812708 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:54:49.814946 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:49.814923 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a653ab\"" Sep 10 15:54:49.823923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:49.823903 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7"] Sep 10 15:54:49.944633 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:49.944594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb28c7b-f592-4558-bd68-90195499484a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7\" (UID: \"5eb28c7b-f592-4558-bd68-90195499484a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:54:49.944836 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:49.944666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvb6g\" (UniqueName: \"kubernetes.io/projected/5eb28c7b-f592-4558-bd68-90195499484a-kube-api-access-rvb6g\") pod \"payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7\" (UID: \"5eb28c7b-f592-4558-bd68-90195499484a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:54:50.046249 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:50.046214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvb6g\" (UniqueName: \"kubernetes.io/projected/5eb28c7b-f592-4558-bd68-90195499484a-kube-api-access-rvb6g\") pod \"payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7\" (UID: \"5eb28c7b-f592-4558-bd68-90195499484a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:54:50.046605 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:50.046563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb28c7b-f592-4558-bd68-90195499484a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7\" (UID: \"5eb28c7b-f592-4558-bd68-90195499484a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:54:50.047274 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:50.047252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb28c7b-f592-4558-bd68-90195499484a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7\" (UID: \"5eb28c7b-f592-4558-bd68-90195499484a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:54:50.054295 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:50.054276 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvb6g\" (UniqueName: \"kubernetes.io/projected/5eb28c7b-f592-4558-bd68-90195499484a-kube-api-access-rvb6g\") pod \"payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7\" (UID: \"5eb28c7b-f592-4558-bd68-90195499484a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:54:50.124234 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:50.124200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:54:50.401124 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:50.401093 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7"] Sep 10 15:54:50.402542 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:54:50.402515 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb28c7b_f592_4558_bd68_90195499484a.slice/crio-0d4ca1c9623130c4e2b317ff37a324aa5956e2a65af5d0ecf297024614100e80 WatchSource:0}: Error finding container 0d4ca1c9623130c4e2b317ff37a324aa5956e2a65af5d0ecf297024614100e80: Status 404 returned error can't find the container with id 0d4ca1c9623130c4e2b317ff37a324aa5956e2a65af5d0ecf297024614100e80 Sep 10 15:54:51.147899 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:51.147862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" event={"ID":"5eb28c7b-f592-4558-bd68-90195499484a","Type":"ContainerStarted","Data":"ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5"} Sep 10 15:54:51.148399 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:51.147938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" event={"ID":"5eb28c7b-f592-4558-bd68-90195499484a","Type":"ContainerStarted","Data":"0d4ca1c9623130c4e2b317ff37a324aa5956e2a65af5d0ecf297024614100e80"} Sep 10 15:54:51.148399 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:51.147980 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:54:51.165405 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:54:51.165360 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" podStartSLOduration=2.165345107 podStartE2EDuration="2.165345107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:51.164272748 +0000 UTC m=+1349.726581967" watchObservedRunningTime="2026-09-10 15:54:51.165345107 +0000 UTC m=+1349.727654326" Sep 10 15:55:02.155793 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:02.155762 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:55:12.272053 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:12.272020 2580 generic.go:356] "Generic (PLEG): container finished" podID="72f068f0-bf96-4710-9a30-a434355e2260" containerID="1b7f1d745e699b37121eefa9afdc6948462e93154c7b2a2afaba6dbfd2ba4713" exitCode=0 Sep 10 15:55:12.272492 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:12.272092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" event={"ID":"72f068f0-bf96-4710-9a30-a434355e2260","Type":"ContainerDied","Data":"1b7f1d745e699b37121eefa9afdc6948462e93154c7b2a2afaba6dbfd2ba4713"} Sep 10 15:55:12.272547 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:12.272492 2580 scope.go:117] "RemoveContainer" containerID="1b7f1d745e699b37121eefa9afdc6948462e93154c7b2a2afaba6dbfd2ba4713" Sep 10 15:55:12.273550 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:12.273525 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:55:12.273618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:12.273589 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:55:12.273659 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:12.273620 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:55:13.284634 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:13.284590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" event={"ID":"72f068f0-bf96-4710-9a30-a434355e2260","Type":"ContainerStarted","Data":"a26ff2ca7b4650bb6a303906e43ce09b82abf19227df5c865e8f9df67597fe89"} Sep 10 15:55:13.528878 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:13.528840 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:55:13.530223 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:13.530191 2580 patch_prober.go:28] interesting pod/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 10 15:55:13.530465 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:13.530437 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 10 15:55:14.528932 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:14.528893 2580 patch_prober.go:28] interesting pod/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 10 15:55:14.529572 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:14.529538 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 10 15:55:15.528989 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:15.528944 2580 patch_prober.go:28] interesting pod/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 10 15:55:15.529521 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:15.529022 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 10 15:55:16.528798 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:16.528761 2580 patch_prober.go:28] interesting pod/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 10 15:55:16.529195 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:16.529163 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 10 15:55:17.108170 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:17.108139 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:55:17.528649 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:17.528606 2580 patch_prober.go:28] interesting pod/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 10 15:55:17.528975 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:17.528946 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 10 15:55:18.528485 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.528448 2580 patch_prober.go:28] interesting pod/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 10 15:55:18.529054 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.529024 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 10 15:55:18.801991 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.801904 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279"] Sep 10 15:55:18.804674 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.804653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:18.807123 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.807101 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-86fc05-dockercfg-m6qck\"" Sep 10 15:55:18.807123 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.807112 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-86fc05\"" Sep 10 15:55:18.815219 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.815194 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279"] Sep 10 15:55:18.831875 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.831844 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb"] Sep 10 15:55:18.834817 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.834792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:18.846190 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.846163 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb"] Sep 10 15:55:18.953222 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.953183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnn88\" (UniqueName: \"kubernetes.io/projected/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-kube-api-access-nnn88\") pod \"payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb\" (UID: \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\") " pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:18.953406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.953257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb\" (UID: \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\") " pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:18.953406 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.953373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690f05e5-bddc-4238-bbad-209427c104ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279\" (UID: \"690f05e5-bddc-4238-bbad-209427c104ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:18.953546 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:18.953499 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wpcz\" (UniqueName: \"kubernetes.io/projected/690f05e5-bddc-4238-bbad-209427c104ce-kube-api-access-7wpcz\") pod \"payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279\" (UID: \"690f05e5-bddc-4238-bbad-209427c104ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:19.054566 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.054468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnn88\" (UniqueName: \"kubernetes.io/projected/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-kube-api-access-nnn88\") pod \"payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb\" (UID: \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\") " pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:19.054566 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.054548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb\" (UID: \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\") " pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:19.054808 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.054574 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690f05e5-bddc-4238-bbad-209427c104ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279\" (UID: \"690f05e5-bddc-4238-bbad-209427c104ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:19.054808 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.054621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wpcz\" (UniqueName: \"kubernetes.io/projected/690f05e5-bddc-4238-bbad-209427c104ce-kube-api-access-7wpcz\") pod \"payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279\" (UID: \"690f05e5-bddc-4238-bbad-209427c104ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:19.055286 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.055263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb\" (UID: \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\") " pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:19.055390 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.055334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690f05e5-bddc-4238-bbad-209427c104ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279\" (UID: \"690f05e5-bddc-4238-bbad-209427c104ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:19.062534 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.062513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnn88\" (UniqueName: \"kubernetes.io/projected/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-kube-api-access-nnn88\") pod \"payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb\" (UID: \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\") " pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:19.062634 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.062551 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wpcz\" (UniqueName: \"kubernetes.io/projected/690f05e5-bddc-4238-bbad-209427c104ce-kube-api-access-7wpcz\") pod \"payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279\" (UID: \"690f05e5-bddc-4238-bbad-209427c104ce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:19.117388 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.117350 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:19.147043 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.147016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:19.395451 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.395408 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279"] Sep 10 15:55:19.396994 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:55:19.396964 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690f05e5_bddc_4238_bbad_209427c104ce.slice/crio-441b705f9c0d7e2790483cf1f528a45ccb2712bb2ba9020aa4f7cd22f8f56816 WatchSource:0}: Error finding container 441b705f9c0d7e2790483cf1f528a45ccb2712bb2ba9020aa4f7cd22f8f56816: Status 404 returned error can't find the container with id 441b705f9c0d7e2790483cf1f528a45ccb2712bb2ba9020aa4f7cd22f8f56816 Sep 10 15:55:19.441407 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.441378 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb"] Sep 10 15:55:19.444832 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:55:19.444807 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86e8300_9e36_4b31_bb27_6ff7c0acb22b.slice/crio-b6fff7edbd275c8e344a9034e8440b7aa57c695b0d2a774a6da8381016aed736 WatchSource:0}: Error finding container b6fff7edbd275c8e344a9034e8440b7aa57c695b0d2a774a6da8381016aed736: Status 404 returned error can't find the container with id b6fff7edbd275c8e344a9034e8440b7aa57c695b0d2a774a6da8381016aed736 Sep 10 15:55:19.533619 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.533584 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:55:19.535212 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:19.535189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:55:20.324466 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:20.324432 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" event={"ID":"c86e8300-9e36-4b31-bb27-6ff7c0acb22b","Type":"ContainerStarted","Data":"cbf2bb0cbad45d549d7c1d3aca926639dc88a5e3ccbc64b2c66958a30cd70f18"} Sep 10 15:55:20.324668 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:20.324498 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:20.324668 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:20.324527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" event={"ID":"c86e8300-9e36-4b31-bb27-6ff7c0acb22b","Type":"ContainerStarted","Data":"b6fff7edbd275c8e344a9034e8440b7aa57c695b0d2a774a6da8381016aed736"} Sep 10 15:55:20.326063 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:20.326037 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" event={"ID":"690f05e5-bddc-4238-bbad-209427c104ce","Type":"ContainerStarted","Data":"cbe6f7279a8dfbe067517306d43afdc1f1dfabb7a23dd678b67a02e85daa78c0"} Sep 10 15:55:20.326192 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:20.326084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" event={"ID":"690f05e5-bddc-4238-bbad-209427c104ce","Type":"ContainerStarted","Data":"441b705f9c0d7e2790483cf1f528a45ccb2712bb2ba9020aa4f7cd22f8f56816"} Sep 10 15:55:20.326192 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:20.326111 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:20.340481 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:20.340441 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" podStartSLOduration=2.3404123119999998 podStartE2EDuration="2.340412312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:20.338926067 +0000 UTC m=+1378.901235257" watchObservedRunningTime="2026-09-10 15:55:20.340412312 +0000 UTC m=+1378.902721502" Sep 10 15:55:20.355214 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:20.355170 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" podStartSLOduration=2.355158024 podStartE2EDuration="2.355158024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:20.352560842 +0000 UTC m=+1378.914870032" watchObservedRunningTime="2026-09-10 15:55:20.355158024 +0000 UTC m=+1378.917467255" Sep 10 15:55:22.235654 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:22.235621 2580 scope.go:117] "RemoveContainer" containerID="7765902f672e995227fabb58c3fe32fefbdadb1a88a0e0d982d945b77f51b30f" Sep 10 15:55:31.334039 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:31.334011 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:55:31.334478 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:31.334072 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:55:45.072226 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.072189 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk"] Sep 10 15:55:45.075789 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.075770 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" Sep 10 15:55:45.078168 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.078142 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:55:45.078278 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.078212 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:55:45.082814 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.082793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk"] Sep 10 15:55:45.206301 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.206267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4a4ced-8d9c-43d4-ad1d-451c4f883122-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk\" (UID: \"5c4a4ced-8d9c-43d4-ad1d-451c4f883122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" Sep 10 15:55:45.307288 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.307260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4a4ced-8d9c-43d4-ad1d-451c4f883122-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk\" (UID: \"5c4a4ced-8d9c-43d4-ad1d-451c4f883122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" Sep 10 15:55:45.307935 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.307905 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4a4ced-8d9c-43d4-ad1d-451c4f883122-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk\" (UID: \"5c4a4ced-8d9c-43d4-ad1d-451c4f883122\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" Sep 10 15:55:45.388930 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.388898 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" Sep 10 15:55:45.658382 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:45.658353 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk"] Sep 10 15:55:45.659954 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:55:45.659925 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4a4ced_8d9c_43d4_ad1d_451c4f883122.slice/crio-d516d4ac5e099eb8e4fec548d705c6ff5bc5b5aa97e3e800bcf447397c80b942 WatchSource:0}: Error finding container d516d4ac5e099eb8e4fec548d705c6ff5bc5b5aa97e3e800bcf447397c80b942: Status 404 returned error can't find the container with id d516d4ac5e099eb8e4fec548d705c6ff5bc5b5aa97e3e800bcf447397c80b942 Sep 10 15:55:46.469828 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:46.469802 2580 generic.go:356] "Generic (PLEG): container finished" podID="5c4a4ced-8d9c-43d4-ad1d-451c4f883122" containerID="1cc42f5f4ebe752578a582c0f43624748e7bf467a3c817c3d1e130d578f36fbf" exitCode=0 Sep 10 15:55:46.470129 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:46.469894 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" event={"ID":"5c4a4ced-8d9c-43d4-ad1d-451c4f883122","Type":"ContainerDied","Data":"1cc42f5f4ebe752578a582c0f43624748e7bf467a3c817c3d1e130d578f36fbf"} Sep 10 15:55:46.470129 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:46.469950 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" event={"ID":"5c4a4ced-8d9c-43d4-ad1d-451c4f883122","Type":"ContainerStarted","Data":"d516d4ac5e099eb8e4fec548d705c6ff5bc5b5aa97e3e800bcf447397c80b942"} Sep 10 15:55:47.621352 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:47.621324 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" Sep 10 15:55:47.731220 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:47.731139 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4a4ced-8d9c-43d4-ad1d-451c4f883122-maas-api-service-ca\") pod \"5c4a4ced-8d9c-43d4-ad1d-451c4f883122\" (UID: \"5c4a4ced-8d9c-43d4-ad1d-451c4f883122\") " Sep 10 15:55:47.731512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:47.731490 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4a4ced-8d9c-43d4-ad1d-451c4f883122-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5c4a4ced-8d9c-43d4-ad1d-451c4f883122" (UID: "5c4a4ced-8d9c-43d4-ad1d-451c4f883122"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:55:47.832754 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:47.832716 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4a4ced-8d9c-43d4-ad1d-451c4f883122-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:55:48.484602 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:48.484572 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" Sep 10 15:55:48.484775 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:48.484606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk" event={"ID":"5c4a4ced-8d9c-43d4-ad1d-451c4f883122","Type":"ContainerDied","Data":"d516d4ac5e099eb8e4fec548d705c6ff5bc5b5aa97e3e800bcf447397c80b942"} Sep 10 15:55:48.484775 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:55:48.484656 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d516d4ac5e099eb8e4fec548d705c6ff5bc5b5aa97e3e800bcf447397c80b942" Sep 10 15:56:00.810102 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:00.810062 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7"] Sep 10 15:56:00.810569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:00.810360 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" podUID="5eb28c7b-f592-4558-bd68-90195499484a" containerName="payload-pre-processing" containerID="cri-o://ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5" gracePeriod=30 Sep 10 15:56:00.965463 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:00.965393 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk"] Sep 10 15:56:00.967670 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:00.967642 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a653ab-6hdxk"] Sep 10 15:56:01.073889 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.073824 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:56:01.263105 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.263072 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvb6g\" (UniqueName: \"kubernetes.io/projected/5eb28c7b-f592-4558-bd68-90195499484a-kube-api-access-rvb6g\") pod \"5eb28c7b-f592-4558-bd68-90195499484a\" (UID: \"5eb28c7b-f592-4558-bd68-90195499484a\") " Sep 10 15:56:01.263298 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.263210 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb28c7b-f592-4558-bd68-90195499484a-plugins-config-volume\") pod \"5eb28c7b-f592-4558-bd68-90195499484a\" (UID: \"5eb28c7b-f592-4558-bd68-90195499484a\") " Sep 10 15:56:01.263611 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.263589 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb28c7b-f592-4558-bd68-90195499484a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5eb28c7b-f592-4558-bd68-90195499484a" (UID: "5eb28c7b-f592-4558-bd68-90195499484a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:01.265129 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.265104 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb28c7b-f592-4558-bd68-90195499484a-kube-api-access-rvb6g" (OuterVolumeSpecName: "kube-api-access-rvb6g") pod "5eb28c7b-f592-4558-bd68-90195499484a" (UID: "5eb28c7b-f592-4558-bd68-90195499484a"). InnerVolumeSpecName "kube-api-access-rvb6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:01.364917 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.364893 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvb6g\" (UniqueName: \"kubernetes.io/projected/5eb28c7b-f592-4558-bd68-90195499484a-kube-api-access-rvb6g\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:01.364917 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.364917 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb28c7b-f592-4558-bd68-90195499484a-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:01.554373 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.554334 2580 generic.go:356] "Generic (PLEG): container finished" podID="5eb28c7b-f592-4558-bd68-90195499484a" containerID="ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5" exitCode=0 Sep 10 15:56:01.554571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.554380 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" Sep 10 15:56:01.554571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.554444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" event={"ID":"5eb28c7b-f592-4558-bd68-90195499484a","Type":"ContainerDied","Data":"ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5"} Sep 10 15:56:01.554571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.554498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7" event={"ID":"5eb28c7b-f592-4558-bd68-90195499484a","Type":"ContainerDied","Data":"0d4ca1c9623130c4e2b317ff37a324aa5956e2a65af5d0ecf297024614100e80"} Sep 10 15:56:01.554571 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.554527 2580 scope.go:117] "RemoveContainer" containerID="ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5" Sep 10 15:56:01.564780 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.564765 2580 scope.go:117] "RemoveContainer" containerID="ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5" Sep 10 15:56:01.565065 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:56:01.565048 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5\": container with ID starting with ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5 not found: ID does not exist" containerID="ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5" Sep 10 15:56:01.565126 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.565073 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5"} err="failed to get container status \"ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5\": rpc error: code = NotFound desc = could not find container \"ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5\": container with ID starting with ac941d39967e183aa5082af812bc8c9dc443e62e024ccc08f02945796fa34cf5 not found: ID does not exist" Sep 10 15:56:01.576086 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.576063 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7"] Sep 10 15:56:01.580121 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:01.580100 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a653ab-76885cc4b-j6zx7"] Sep 10 15:56:02.028988 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:02.028948 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c4a4ced-8d9c-43d4-ad1d-451c4f883122" path="/var/lib/kubelet/pods/5c4a4ced-8d9c-43d4-ad1d-451c4f883122/volumes" Sep 10 15:56:02.029534 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:02.029515 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eb28c7b-f592-4558-bd68-90195499484a" path="/var/lib/kubelet/pods/5eb28c7b-f592-4558-bd68-90195499484a/volumes" Sep 10 15:56:07.393583 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.393545 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8"] Sep 10 15:56:07.394147 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.393847 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" containerID="cri-o://a26ff2ca7b4650bb6a303906e43ce09b82abf19227df5c865e8f9df67597fe89" gracePeriod=30 Sep 10 15:56:07.771111 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.771034 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj"] Sep 10 15:56:07.771504 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.771490 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb28c7b-f592-4558-bd68-90195499484a" containerName="payload-pre-processing" Sep 10 15:56:07.771504 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.771505 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb28c7b-f592-4558-bd68-90195499484a" containerName="payload-pre-processing" Sep 10 15:56:07.771601 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.771533 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c4a4ced-8d9c-43d4-ad1d-451c4f883122" containerName="revoke-keys" Sep 10 15:56:07.771601 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.771540 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c4a4ced-8d9c-43d4-ad1d-451c4f883122" containerName="revoke-keys" Sep 10 15:56:07.771665 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.771611 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb28c7b-f592-4558-bd68-90195499484a" containerName="payload-pre-processing" Sep 10 15:56:07.771665 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.771620 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c4a4ced-8d9c-43d4-ad1d-451c4f883122" containerName="revoke-keys" Sep 10 15:56:07.775152 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.775136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" Sep 10 15:56:07.777281 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.777261 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:56:07.777378 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.777263 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:56:07.781289 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.781267 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj"] Sep 10 15:56:07.823919 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.823879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe97908-91da-4c7e-a64d-8b603ba7a080-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj\" (UID: \"6fe97908-91da-4c7e-a64d-8b603ba7a080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" Sep 10 15:56:07.924831 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.924800 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe97908-91da-4c7e-a64d-8b603ba7a080-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj\" (UID: \"6fe97908-91da-4c7e-a64d-8b603ba7a080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" Sep 10 15:56:07.925514 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:07.925494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe97908-91da-4c7e-a64d-8b603ba7a080-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj\" (UID: \"6fe97908-91da-4c7e-a64d-8b603ba7a080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" Sep 10 15:56:08.087959 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:08.087865 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" Sep 10 15:56:08.359486 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:08.359459 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj"] Sep 10 15:56:08.361753 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:56:08.361724 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe97908_91da_4c7e_a64d_8b603ba7a080.slice/crio-2fa9698dbbb3b0c598f77bf8d2b7f5dcd6a6024e9306969c0b78ca06d4c71cad WatchSource:0}: Error finding container 2fa9698dbbb3b0c598f77bf8d2b7f5dcd6a6024e9306969c0b78ca06d4c71cad: Status 404 returned error can't find the container with id 2fa9698dbbb3b0c598f77bf8d2b7f5dcd6a6024e9306969c0b78ca06d4c71cad Sep 10 15:56:08.597726 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:08.597693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" event={"ID":"6fe97908-91da-4c7e-a64d-8b603ba7a080","Type":"ContainerStarted","Data":"2fa9698dbbb3b0c598f77bf8d2b7f5dcd6a6024e9306969c0b78ca06d4c71cad"} Sep 10 15:56:09.604755 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:09.604725 2580 generic.go:356] "Generic (PLEG): container finished" podID="6fe97908-91da-4c7e-a64d-8b603ba7a080" containerID="bf9347470feacbce5cfda92f08ddc242395e1af7b8f5571a85470330e9cb7be7" exitCode=0 Sep 10 15:56:09.605116 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:09.604803 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" event={"ID":"6fe97908-91da-4c7e-a64d-8b603ba7a080","Type":"ContainerDied","Data":"bf9347470feacbce5cfda92f08ddc242395e1af7b8f5571a85470330e9cb7be7"} Sep 10 15:56:10.753254 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:10.753227 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" Sep 10 15:56:10.854581 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:10.854534 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe97908-91da-4c7e-a64d-8b603ba7a080-maas-api-service-ca\") pod \"6fe97908-91da-4c7e-a64d-8b603ba7a080\" (UID: \"6fe97908-91da-4c7e-a64d-8b603ba7a080\") " Sep 10 15:56:10.854931 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:10.854906 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe97908-91da-4c7e-a64d-8b603ba7a080-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fe97908-91da-4c7e-a64d-8b603ba7a080" (UID: "6fe97908-91da-4c7e-a64d-8b603ba7a080"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:10.955978 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:10.955894 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fe97908-91da-4c7e-a64d-8b603ba7a080-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:11.617334 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:11.617307 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" Sep 10 15:56:11.617515 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:11.617346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj" event={"ID":"6fe97908-91da-4c7e-a64d-8b603ba7a080","Type":"ContainerDied","Data":"2fa9698dbbb3b0c598f77bf8d2b7f5dcd6a6024e9306969c0b78ca06d4c71cad"} Sep 10 15:56:11.617515 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:11.617388 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fa9698dbbb3b0c598f77bf8d2b7f5dcd6a6024e9306969c0b78ca06d4c71cad" Sep 10 15:56:12.626753 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.626712 2580 generic.go:356] "Generic (PLEG): container finished" podID="72f068f0-bf96-4710-9a30-a434355e2260" containerID="a26ff2ca7b4650bb6a303906e43ce09b82abf19227df5c865e8f9df67597fe89" exitCode=0 Sep 10 15:56:12.627114 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.626780 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" event={"ID":"72f068f0-bf96-4710-9a30-a434355e2260","Type":"ContainerDied","Data":"a26ff2ca7b4650bb6a303906e43ce09b82abf19227df5c865e8f9df67597fe89"} Sep 10 15:56:12.627114 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.626841 2580 scope.go:117] "RemoveContainer" containerID="1b7f1d745e699b37121eefa9afdc6948462e93154c7b2a2afaba6dbfd2ba4713" Sep 10 15:56:12.651386 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.651371 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:56:12.674260 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674206 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f068f0-bf96-4710-9a30-a434355e2260-istio-podinfo\") pod \"72f068f0-bf96-4710-9a30-a434355e2260\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " Sep 10 15:56:12.674260 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674254 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f068f0-bf96-4710-9a30-a434355e2260-istiod-ca-cert\") pod \"72f068f0-bf96-4710-9a30-a434355e2260\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " Sep 10 15:56:12.674441 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674281 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-istio-token\") pod \"72f068f0-bf96-4710-9a30-a434355e2260\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " Sep 10 15:56:12.674441 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674319 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-envoy\") pod \"72f068f0-bf96-4710-9a30-a434355e2260\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " Sep 10 15:56:12.674556 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674476 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-socket\") pod \"72f068f0-bf96-4710-9a30-a434355e2260\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " Sep 10 15:56:12.674556 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674545 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-certs\") pod \"72f068f0-bf96-4710-9a30-a434355e2260\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " Sep 10 15:56:12.674670 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674606 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-credential-socket\") pod \"72f068f0-bf96-4710-9a30-a434355e2260\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " Sep 10 15:56:12.674670 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674651 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-data\") pod \"72f068f0-bf96-4710-9a30-a434355e2260\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " Sep 10 15:56:12.674777 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674677 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f068f0-bf96-4710-9a30-a434355e2260-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72f068f0-bf96-4710-9a30-a434355e2260" (UID: "72f068f0-bf96-4710-9a30-a434355e2260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:12.674777 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674677 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72f068f0-bf96-4710-9a30-a434355e2260" (UID: "72f068f0-bf96-4710-9a30-a434355e2260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:12.674777 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674743 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l2p5\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-kube-api-access-8l2p5\") pod \"72f068f0-bf96-4710-9a30-a434355e2260\" (UID: \"72f068f0-bf96-4710-9a30-a434355e2260\") " Sep 10 15:56:12.674940 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674840 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72f068f0-bf96-4710-9a30-a434355e2260" (UID: "72f068f0-bf96-4710-9a30-a434355e2260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:12.674940 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674875 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72f068f0-bf96-4710-9a30-a434355e2260" (UID: "72f068f0-bf96-4710-9a30-a434355e2260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:12.674940 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.674929 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-data" (OuterVolumeSpecName: "istio-data") pod "72f068f0-bf96-4710-9a30-a434355e2260" (UID: "72f068f0-bf96-4710-9a30-a434355e2260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:12.675198 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.675171 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f068f0-bf96-4710-9a30-a434355e2260-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:12.675198 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.675199 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:12.675376 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.675211 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:12.675376 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.675219 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:12.675376 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.675226 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:12.676569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.676535 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f068f0-bf96-4710-9a30-a434355e2260-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72f068f0-bf96-4710-9a30-a434355e2260" (UID: "72f068f0-bf96-4710-9a30-a434355e2260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:56:12.676854 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.676823 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72f068f0-bf96-4710-9a30-a434355e2260" (UID: "72f068f0-bf96-4710-9a30-a434355e2260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:12.676955 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.676871 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-istio-token" (OuterVolumeSpecName: "istio-token") pod "72f068f0-bf96-4710-9a30-a434355e2260" (UID: "72f068f0-bf96-4710-9a30-a434355e2260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:12.677002 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.676970 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-kube-api-access-8l2p5" (OuterVolumeSpecName: "kube-api-access-8l2p5") pod "72f068f0-bf96-4710-9a30-a434355e2260" (UID: "72f068f0-bf96-4710-9a30-a434355e2260"). InnerVolumeSpecName "kube-api-access-8l2p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:12.776115 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.776089 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f068f0-bf96-4710-9a30-a434355e2260-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:12.776115 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.776114 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:12.776281 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.776122 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f068f0-bf96-4710-9a30-a434355e2260-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:12.776281 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:12.776130 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l2p5\" (UniqueName: \"kubernetes.io/projected/72f068f0-bf96-4710-9a30-a434355e2260-kube-api-access-8l2p5\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:13.634528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:13.634418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" event={"ID":"72f068f0-bf96-4710-9a30-a434355e2260","Type":"ContainerDied","Data":"438e1b9f0f545f536caa2d0b2ad2e31828fdec5c9b23cc3eee368da382672648"} Sep 10 15:56:13.634528 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:13.634504 2580 scope.go:117] "RemoveContainer" containerID="a26ff2ca7b4650bb6a303906e43ce09b82abf19227df5c865e8f9df67597fe89" Sep 10 15:56:13.635000 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:13.634452 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8" Sep 10 15:56:13.666638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:13.666603 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8"] Sep 10 15:56:13.670456 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:13.670435 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a653ab-openshift-default-975cb8884-qlbq8"] Sep 10 15:56:14.031569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:14.031492 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72f068f0-bf96-4710-9a30-a434355e2260" path="/var/lib/kubelet/pods/72f068f0-bf96-4710-9a30-a434355e2260/volumes" Sep 10 15:56:22.267464 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:22.267418 2580 scope.go:117] "RemoveContainer" containerID="510244a10c5131da33522f2808a8525af2fb38a7db818b3a99dc2bb2a8db52eb" Sep 10 15:56:22.275796 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:22.275780 2580 scope.go:117] "RemoveContainer" containerID="5f607de9fbccae484c91f181f61f7a141a928fa3fc013ae2a73a61556458e370" Sep 10 15:56:22.283210 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:22.283186 2580 scope.go:117] "RemoveContainer" containerID="556c8cf1c85f8f8e38b1fe1bb7edc07886789f656001f7af596bec57092afe9f" Sep 10 15:56:23.535853 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.535814 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb"] Sep 10 15:56:23.536253 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.536114 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" podUID="c86e8300-9e36-4b31-bb27-6ff7c0acb22b" containerName="payload-processing" containerID="cri-o://cbf2bb0cbad45d549d7c1d3aca926639dc88a5e3ccbc64b2c66958a30cd70f18" gracePeriod=30 Sep 10 15:56:23.539885 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.539848 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279"] Sep 10 15:56:23.540123 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.540076 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" podUID="690f05e5-bddc-4238-bbad-209427c104ce" containerName="payload-pre-processing" containerID="cri-o://cbe6f7279a8dfbe067517306d43afdc1f1dfabb7a23dd678b67a02e85daa78c0" gracePeriod=30 Sep 10 15:56:23.688061 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.688020 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj"] Sep 10 15:56:23.690316 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.690291 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-86fc05-t2dfj"] Sep 10 15:56:23.691377 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.691350 2580 generic.go:356] "Generic (PLEG): container finished" podID="c86e8300-9e36-4b31-bb27-6ff7c0acb22b" containerID="cbf2bb0cbad45d549d7c1d3aca926639dc88a5e3ccbc64b2c66958a30cd70f18" exitCode=0 Sep 10 15:56:23.691494 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.691404 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" event={"ID":"c86e8300-9e36-4b31-bb27-6ff7c0acb22b","Type":"ContainerDied","Data":"cbf2bb0cbad45d549d7c1d3aca926639dc88a5e3ccbc64b2c66958a30cd70f18"} Sep 10 15:56:23.693792 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.693771 2580 generic.go:356] "Generic (PLEG): container finished" podID="690f05e5-bddc-4238-bbad-209427c104ce" containerID="cbe6f7279a8dfbe067517306d43afdc1f1dfabb7a23dd678b67a02e85daa78c0" exitCode=0 Sep 10 15:56:23.693894 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.693801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" event={"ID":"690f05e5-bddc-4238-bbad-209427c104ce","Type":"ContainerDied","Data":"cbe6f7279a8dfbe067517306d43afdc1f1dfabb7a23dd678b67a02e85daa78c0"} Sep 10 15:56:23.833565 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.833538 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:56:23.836789 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.836769 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:56:23.984474 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.984412 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-plugins-config-volume\") pod \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\" (UID: \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\") " Sep 10 15:56:23.984669 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.984500 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wpcz\" (UniqueName: \"kubernetes.io/projected/690f05e5-bddc-4238-bbad-209427c104ce-kube-api-access-7wpcz\") pod \"690f05e5-bddc-4238-bbad-209427c104ce\" (UID: \"690f05e5-bddc-4238-bbad-209427c104ce\") " Sep 10 15:56:23.984669 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.984534 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690f05e5-bddc-4238-bbad-209427c104ce-plugins-config-volume\") pod \"690f05e5-bddc-4238-bbad-209427c104ce\" (UID: \"690f05e5-bddc-4238-bbad-209427c104ce\") " Sep 10 15:56:23.984669 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.984609 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnn88\" (UniqueName: \"kubernetes.io/projected/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-kube-api-access-nnn88\") pod \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\" (UID: \"c86e8300-9e36-4b31-bb27-6ff7c0acb22b\") " Sep 10 15:56:23.984953 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.984919 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c86e8300-9e36-4b31-bb27-6ff7c0acb22b" (UID: "c86e8300-9e36-4b31-bb27-6ff7c0acb22b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:23.985048 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.984948 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690f05e5-bddc-4238-bbad-209427c104ce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "690f05e5-bddc-4238-bbad-209427c104ce" (UID: "690f05e5-bddc-4238-bbad-209427c104ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:23.986784 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.986760 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690f05e5-bddc-4238-bbad-209427c104ce-kube-api-access-7wpcz" (OuterVolumeSpecName: "kube-api-access-7wpcz") pod "690f05e5-bddc-4238-bbad-209427c104ce" (UID: "690f05e5-bddc-4238-bbad-209427c104ce"). InnerVolumeSpecName "kube-api-access-7wpcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:23.986784 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:23.986768 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-kube-api-access-nnn88" (OuterVolumeSpecName: "kube-api-access-nnn88") pod "c86e8300-9e36-4b31-bb27-6ff7c0acb22b" (UID: "c86e8300-9e36-4b31-bb27-6ff7c0acb22b"). InnerVolumeSpecName "kube-api-access-nnn88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:24.029155 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.029125 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe97908-91da-4c7e-a64d-8b603ba7a080" path="/var/lib/kubelet/pods/6fe97908-91da-4c7e-a64d-8b603ba7a080/volumes" Sep 10 15:56:24.086014 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.085919 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:24.086014 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.085950 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wpcz\" (UniqueName: \"kubernetes.io/projected/690f05e5-bddc-4238-bbad-209427c104ce-kube-api-access-7wpcz\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:24.086014 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.085966 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690f05e5-bddc-4238-bbad-209427c104ce-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:24.086014 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.085981 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnn88\" (UniqueName: \"kubernetes.io/projected/c86e8300-9e36-4b31-bb27-6ff7c0acb22b-kube-api-access-nnn88\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:24.700001 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.699974 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" Sep 10 15:56:24.700001 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.699993 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb" event={"ID":"c86e8300-9e36-4b31-bb27-6ff7c0acb22b","Type":"ContainerDied","Data":"b6fff7edbd275c8e344a9034e8440b7aa57c695b0d2a774a6da8381016aed736"} Sep 10 15:56:24.700547 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.700063 2580 scope.go:117] "RemoveContainer" containerID="cbf2bb0cbad45d549d7c1d3aca926639dc88a5e3ccbc64b2c66958a30cd70f18" Sep 10 15:56:24.701398 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.701375 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" event={"ID":"690f05e5-bddc-4238-bbad-209427c104ce","Type":"ContainerDied","Data":"441b705f9c0d7e2790483cf1f528a45ccb2712bb2ba9020aa4f7cd22f8f56816"} Sep 10 15:56:24.701479 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.701407 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279" Sep 10 15:56:24.709603 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.709586 2580 scope.go:117] "RemoveContainer" containerID="cbe6f7279a8dfbe067517306d43afdc1f1dfabb7a23dd678b67a02e85daa78c0" Sep 10 15:56:24.719125 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.719082 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279"] Sep 10 15:56:24.722931 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.722910 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86fc05-65d7c9f9d6-5m279"] Sep 10 15:56:24.735079 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.735056 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb"] Sep 10 15:56:24.738302 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:24.738280 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86fc05-5896945d85-ps9vb"] Sep 10 15:56:26.029591 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:26.029554 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="690f05e5-bddc-4238-bbad-209427c104ce" path="/var/lib/kubelet/pods/690f05e5-bddc-4238-bbad-209427c104ce/volumes" Sep 10 15:56:26.030130 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:26.030084 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c86e8300-9e36-4b31-bb27-6ff7c0acb22b" path="/var/lib/kubelet/pods/c86e8300-9e36-4b31-bb27-6ff7c0acb22b/volumes" Sep 10 15:56:30.289195 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289164 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8"] Sep 10 15:56:30.289687 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289671 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" Sep 10 15:56:30.289687 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289688 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" Sep 10 15:56:30.289764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289698 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" Sep 10 15:56:30.289764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289704 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" Sep 10 15:56:30.289764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289712 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c86e8300-9e36-4b31-bb27-6ff7c0acb22b" containerName="payload-processing" Sep 10 15:56:30.289764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289718 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c86e8300-9e36-4b31-bb27-6ff7c0acb22b" containerName="payload-processing" Sep 10 15:56:30.289764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289726 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="690f05e5-bddc-4238-bbad-209427c104ce" containerName="payload-pre-processing" Sep 10 15:56:30.289764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289731 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="690f05e5-bddc-4238-bbad-209427c104ce" containerName="payload-pre-processing" Sep 10 15:56:30.289764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289751 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe97908-91da-4c7e-a64d-8b603ba7a080" containerName="revoke-keys" Sep 10 15:56:30.289764 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289757 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe97908-91da-4c7e-a64d-8b603ba7a080" containerName="revoke-keys" Sep 10 15:56:30.290030 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289812 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c86e8300-9e36-4b31-bb27-6ff7c0acb22b" containerName="payload-processing" Sep 10 15:56:30.290030 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289825 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe97908-91da-4c7e-a64d-8b603ba7a080" containerName="revoke-keys" Sep 10 15:56:30.290030 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289832 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" Sep 10 15:56:30.290030 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.289841 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="690f05e5-bddc-4238-bbad-209427c104ce" containerName="payload-pre-processing" Sep 10 15:56:30.292970 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.292954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" Sep 10 15:56:30.295204 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.295184 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:56:30.295304 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.295189 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:56:30.298940 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.298917 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8"] Sep 10 15:56:30.447334 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.447297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d25c32-bc11-4d6e-8ff4-2a144ea33f93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8\" (UID: \"49d25c32-bc11-4d6e-8ff4-2a144ea33f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" Sep 10 15:56:30.548460 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.548329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d25c32-bc11-4d6e-8ff4-2a144ea33f93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8\" (UID: \"49d25c32-bc11-4d6e-8ff4-2a144ea33f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" Sep 10 15:56:30.549000 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.548977 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d25c32-bc11-4d6e-8ff4-2a144ea33f93-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8\" (UID: \"49d25c32-bc11-4d6e-8ff4-2a144ea33f93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" Sep 10 15:56:30.605293 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.605257 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" Sep 10 15:56:30.874610 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:30.874583 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8"] Sep 10 15:56:30.876500 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:56:30.876473 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d25c32_bc11_4d6e_8ff4_2a144ea33f93.slice/crio-89c4b81d86c70e94842ffeddafbfd2105ca426b32ae81f79c5442c4bfba7b72e WatchSource:0}: Error finding container 89c4b81d86c70e94842ffeddafbfd2105ca426b32ae81f79c5442c4bfba7b72e: Status 404 returned error can't find the container with id 89c4b81d86c70e94842ffeddafbfd2105ca426b32ae81f79c5442c4bfba7b72e Sep 10 15:56:31.739689 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:31.739658 2580 generic.go:356] "Generic (PLEG): container finished" podID="49d25c32-bc11-4d6e-8ff4-2a144ea33f93" containerID="d3431925807dd55e22e4718db92d8b9b271dd33fb3c7bb4c0ed0959caa60ca30" exitCode=0 Sep 10 15:56:31.740022 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:31.739742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" event={"ID":"49d25c32-bc11-4d6e-8ff4-2a144ea33f93","Type":"ContainerDied","Data":"d3431925807dd55e22e4718db92d8b9b271dd33fb3c7bb4c0ed0959caa60ca30"} Sep 10 15:56:31.740022 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:31.739793 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" event={"ID":"49d25c32-bc11-4d6e-8ff4-2a144ea33f93","Type":"ContainerStarted","Data":"89c4b81d86c70e94842ffeddafbfd2105ca426b32ae81f79c5442c4bfba7b72e"} Sep 10 15:56:32.873872 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:32.873839 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" Sep 10 15:56:32.972462 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:32.972414 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d25c32-bc11-4d6e-8ff4-2a144ea33f93-maas-api-service-ca\") pod \"49d25c32-bc11-4d6e-8ff4-2a144ea33f93\" (UID: \"49d25c32-bc11-4d6e-8ff4-2a144ea33f93\") " Sep 10 15:56:32.972695 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:32.972674 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d25c32-bc11-4d6e-8ff4-2a144ea33f93-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "49d25c32-bc11-4d6e-8ff4-2a144ea33f93" (UID: "49d25c32-bc11-4d6e-8ff4-2a144ea33f93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:33.073595 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:33.073526 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d25c32-bc11-4d6e-8ff4-2a144ea33f93-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:33.751633 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:33.751601 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" event={"ID":"49d25c32-bc11-4d6e-8ff4-2a144ea33f93","Type":"ContainerDied","Data":"89c4b81d86c70e94842ffeddafbfd2105ca426b32ae81f79c5442c4bfba7b72e"} Sep 10 15:56:33.751633 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:33.751629 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8" Sep 10 15:56:33.751833 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:33.751646 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89c4b81d86c70e94842ffeddafbfd2105ca426b32ae81f79c5442c4bfba7b72e" Sep 10 15:56:41.055640 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.055593 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9"] Sep 10 15:56:41.056263 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.055842 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" podUID="e13767c2-09ca-402e-a8ca-3bb6ff810dc2" containerName="payload-processing" containerID="cri-o://bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433" gracePeriod=30 Sep 10 15:56:41.254440 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.253960 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8"] Sep 10 15:56:41.257002 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.256976 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8e3dc2-gzgp8"] Sep 10 15:56:41.310398 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.310345 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:56:41.353156 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.353128 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjzjx\" (UniqueName: \"kubernetes.io/projected/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-kube-api-access-qjzjx\") pod \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\" (UID: \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\") " Sep 10 15:56:41.353327 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.353210 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-plugins-config-volume\") pod \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\" (UID: \"e13767c2-09ca-402e-a8ca-3bb6ff810dc2\") " Sep 10 15:56:41.353643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.353618 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e13767c2-09ca-402e-a8ca-3bb6ff810dc2" (UID: "e13767c2-09ca-402e-a8ca-3bb6ff810dc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:41.355495 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.355467 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-kube-api-access-qjzjx" (OuterVolumeSpecName: "kube-api-access-qjzjx") pod "e13767c2-09ca-402e-a8ca-3bb6ff810dc2" (UID: "e13767c2-09ca-402e-a8ca-3bb6ff810dc2"). InnerVolumeSpecName "kube-api-access-qjzjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:41.454675 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.454645 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzjx\" (UniqueName: \"kubernetes.io/projected/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-kube-api-access-qjzjx\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:41.454675 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.454670 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13767c2-09ca-402e-a8ca-3bb6ff810dc2-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:41.800105 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.800066 2580 generic.go:356] "Generic (PLEG): container finished" podID="e13767c2-09ca-402e-a8ca-3bb6ff810dc2" containerID="bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433" exitCode=0 Sep 10 15:56:41.800105 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.800106 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" Sep 10 15:56:41.800344 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.800148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" event={"ID":"e13767c2-09ca-402e-a8ca-3bb6ff810dc2","Type":"ContainerDied","Data":"bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433"} Sep 10 15:56:41.800344 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.800207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9" event={"ID":"e13767c2-09ca-402e-a8ca-3bb6ff810dc2","Type":"ContainerDied","Data":"4a703cd9e1a9e619233796db052ffde2dc9856afb03ad52337eb15c84755ab42"} Sep 10 15:56:41.800344 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.800230 2580 scope.go:117] "RemoveContainer" containerID="bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433" Sep 10 15:56:41.809562 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.809547 2580 scope.go:117] "RemoveContainer" containerID="bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433" Sep 10 15:56:41.809816 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:56:41.809798 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433\": container with ID starting with bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433 not found: ID does not exist" containerID="bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433" Sep 10 15:56:41.809861 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.809825 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433"} err="failed to get container status \"bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433\": rpc error: code = NotFound desc = could not find container \"bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433\": container with ID starting with bb7a5603db4a527da3b154d4e0b9a69eefd1bf8852587d49ccdd8e36079f6433 not found: ID does not exist" Sep 10 15:56:41.821725 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.821703 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9"] Sep 10 15:56:41.827864 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:41.827840 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8e3dc2-849c65747-fncx9"] Sep 10 15:56:42.028878 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:42.028838 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49d25c32-bc11-4d6e-8ff4-2a144ea33f93" path="/var/lib/kubelet/pods/49d25c32-bc11-4d6e-8ff4-2a144ea33f93/volumes" Sep 10 15:56:42.029304 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:42.029285 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e13767c2-09ca-402e-a8ca-3bb6ff810dc2" path="/var/lib/kubelet/pods/e13767c2-09ca-402e-a8ca-3bb6ff810dc2/volumes" Sep 10 15:56:48.096227 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.096189 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62"] Sep 10 15:56:48.096672 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.096658 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13767c2-09ca-402e-a8ca-3bb6ff810dc2" containerName="payload-processing" Sep 10 15:56:48.096672 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.096672 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13767c2-09ca-402e-a8ca-3bb6ff810dc2" containerName="payload-processing" Sep 10 15:56:48.096756 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.096689 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d25c32-bc11-4d6e-8ff4-2a144ea33f93" containerName="revoke-keys" Sep 10 15:56:48.096756 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.096695 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d25c32-bc11-4d6e-8ff4-2a144ea33f93" containerName="revoke-keys" Sep 10 15:56:48.096835 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.096762 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d25c32-bc11-4d6e-8ff4-2a144ea33f93" containerName="revoke-keys" Sep 10 15:56:48.096835 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.096772 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f068f0-bf96-4710-9a30-a434355e2260" containerName="istio-proxy" Sep 10 15:56:48.096835 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.096782 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13767c2-09ca-402e-a8ca-3bb6ff810dc2" containerName="payload-processing" Sep 10 15:56:48.099593 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.099578 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" Sep 10 15:56:48.101801 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.101778 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:56:48.101944 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.101908 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:56:48.107983 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.107956 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62"] Sep 10 15:56:48.118966 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.118938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03cdabd-239c-4eb9-9b38-aa1e0f479d33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62\" (UID: \"c03cdabd-239c-4eb9-9b38-aa1e0f479d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" Sep 10 15:56:48.220016 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.219982 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03cdabd-239c-4eb9-9b38-aa1e0f479d33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62\" (UID: \"c03cdabd-239c-4eb9-9b38-aa1e0f479d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" Sep 10 15:56:48.220710 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.220692 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03cdabd-239c-4eb9-9b38-aa1e0f479d33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62\" (UID: \"c03cdabd-239c-4eb9-9b38-aa1e0f479d33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" Sep 10 15:56:48.412629 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.412576 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" Sep 10 15:56:48.683860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.683830 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62"] Sep 10 15:56:48.686187 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:56:48.686159 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03cdabd_239c_4eb9_9b38_aa1e0f479d33.slice/crio-a0ceec81e39508f903ea8f11feafe6f2095aff0ccdc00eefd684de5eb4b079bd WatchSource:0}: Error finding container a0ceec81e39508f903ea8f11feafe6f2095aff0ccdc00eefd684de5eb4b079bd: Status 404 returned error can't find the container with id a0ceec81e39508f903ea8f11feafe6f2095aff0ccdc00eefd684de5eb4b079bd Sep 10 15:56:48.840235 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:48.840203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" event={"ID":"c03cdabd-239c-4eb9-9b38-aa1e0f479d33","Type":"ContainerStarted","Data":"a0ceec81e39508f903ea8f11feafe6f2095aff0ccdc00eefd684de5eb4b079bd"} Sep 10 15:56:49.846446 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:49.846340 2580 generic.go:356] "Generic (PLEG): container finished" podID="c03cdabd-239c-4eb9-9b38-aa1e0f479d33" containerID="11bc5ed8105a5b6c77b151e42d2ccca2c6e7211850758649baa8bd1aa6095e9e" exitCode=0 Sep 10 15:56:49.846860 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:49.846438 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" event={"ID":"c03cdabd-239c-4eb9-9b38-aa1e0f479d33","Type":"ContainerDied","Data":"11bc5ed8105a5b6c77b151e42d2ccca2c6e7211850758649baa8bd1aa6095e9e"} Sep 10 15:56:50.980681 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:50.980376 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" Sep 10 15:56:51.043825 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:51.043795 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03cdabd-239c-4eb9-9b38-aa1e0f479d33-maas-api-service-ca\") pod \"c03cdabd-239c-4eb9-9b38-aa1e0f479d33\" (UID: \"c03cdabd-239c-4eb9-9b38-aa1e0f479d33\") " Sep 10 15:56:51.044170 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:51.044146 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03cdabd-239c-4eb9-9b38-aa1e0f479d33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c03cdabd-239c-4eb9-9b38-aa1e0f479d33" (UID: "c03cdabd-239c-4eb9-9b38-aa1e0f479d33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:51.145364 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:51.145332 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03cdabd-239c-4eb9-9b38-aa1e0f479d33-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:51.857793 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:51.857767 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" Sep 10 15:56:51.857977 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:51.857805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62" event={"ID":"c03cdabd-239c-4eb9-9b38-aa1e0f479d33","Type":"ContainerDied","Data":"a0ceec81e39508f903ea8f11feafe6f2095aff0ccdc00eefd684de5eb4b079bd"} Sep 10 15:56:51.857977 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:51.857849 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0ceec81e39508f903ea8f11feafe6f2095aff0ccdc00eefd684de5eb4b079bd" Sep 10 15:56:54.170850 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.170813 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s"] Sep 10 15:56:54.171417 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.171228 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03cdabd-239c-4eb9-9b38-aa1e0f479d33" containerName="revoke-keys" Sep 10 15:56:54.171417 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.171244 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03cdabd-239c-4eb9-9b38-aa1e0f479d33" containerName="revoke-keys" Sep 10 15:56:54.171417 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.171314 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03cdabd-239c-4eb9-9b38-aa1e0f479d33" containerName="revoke-keys" Sep 10 15:56:54.173972 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.173951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.176354 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.176335 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a85a91-openshift-default-dockercfg-z7468\"" Sep 10 15:56:54.186557 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.186535 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s"] Sep 10 15:56:54.269960 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.269924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-envoy\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.270134 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.270043 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-data\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.270134 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.270100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istiod-ca-cert\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.270235 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.270138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-socket\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.270235 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.270176 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvll\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-kube-api-access-dpvll\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.270235 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.270219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-credential-socket\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.270373 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.270246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-podinfo\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.270373 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.270296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-certs\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.270373 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.270339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-token\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.371829 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.371794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-certs\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.371829 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.371827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-token\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.371857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-envoy\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.371900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-data\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.371937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istiod-ca-cert\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.371967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-socket\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.371991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvll\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-kube-api-access-dpvll\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.372024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-credential-socket\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372060 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.372051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-podinfo\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372405 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.372244 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-certs\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372484 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.372396 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-data\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372550 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.372485 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-socket\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372638 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.372618 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-credential-socket\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.372678 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.372664 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istiod-ca-cert\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.374194 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.374176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-envoy\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.374512 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.374495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-podinfo\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.380462 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.380405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-token\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.380643 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.380625 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvll\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-kube-api-access-dpvll\") pod \"e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.484193 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.484100 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:54.764377 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.764353 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s"] Sep 10 15:56:54.766582 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:56:54.766555 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdc0b48_61e0_4eda_b79b_b4c2a73c6d0b.slice/crio-1f0c6cf041a9747bcfccb7d5d42e8d16011ed086c1f158c5b76fa7f133efcc39 WatchSource:0}: Error finding container 1f0c6cf041a9747bcfccb7d5d42e8d16011ed086c1f158c5b76fa7f133efcc39: Status 404 returned error can't find the container with id 1f0c6cf041a9747bcfccb7d5d42e8d16011ed086c1f158c5b76fa7f133efcc39 Sep 10 15:56:54.768814 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.768785 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:56:54.768905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.768846 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:56:54.768905 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.768875 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:56:54.877250 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.877213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" event={"ID":"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b","Type":"ContainerStarted","Data":"2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d"} Sep 10 15:56:54.877451 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.877274 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" event={"ID":"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b","Type":"ContainerStarted","Data":"1f0c6cf041a9747bcfccb7d5d42e8d16011ed086c1f158c5b76fa7f133efcc39"} Sep 10 15:56:54.897926 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:54.897878 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" podStartSLOduration=0.897861297 podStartE2EDuration="897.861297ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:54.895208359 +0000 UTC m=+1473.457517574" watchObservedRunningTime="2026-09-10 15:56:54.897861297 +0000 UTC m=+1473.460170486" Sep 10 15:56:55.484516 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:55.484481 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:55.489574 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:55.489549 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:55.883279 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:55.883249 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:55.884175 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:55.884155 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:56:58.604229 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.604195 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln"] Sep 10 15:56:58.607064 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.607037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:56:58.610935 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.610915 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a85a91\"" Sep 10 15:56:58.610935 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.610926 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a85a91-dockercfg-k869s\"" Sep 10 15:56:58.627100 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.627073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln"] Sep 10 15:56:58.712923 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.712880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77be5ae-4aa7-4b53-a6c2-4675749db923-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln\" (UID: \"a77be5ae-4aa7-4b53-a6c2-4675749db923\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:56:58.713133 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.712971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptsmh\" (UniqueName: \"kubernetes.io/projected/a77be5ae-4aa7-4b53-a6c2-4675749db923-kube-api-access-ptsmh\") pod \"payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln\" (UID: \"a77be5ae-4aa7-4b53-a6c2-4675749db923\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:56:58.813758 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.813722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptsmh\" (UniqueName: \"kubernetes.io/projected/a77be5ae-4aa7-4b53-a6c2-4675749db923-kube-api-access-ptsmh\") pod \"payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln\" (UID: \"a77be5ae-4aa7-4b53-a6c2-4675749db923\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:56:58.813933 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.813912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77be5ae-4aa7-4b53-a6c2-4675749db923-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln\" (UID: \"a77be5ae-4aa7-4b53-a6c2-4675749db923\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:56:58.814635 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.814613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77be5ae-4aa7-4b53-a6c2-4675749db923-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln\" (UID: \"a77be5ae-4aa7-4b53-a6c2-4675749db923\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:56:58.822724 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.822696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptsmh\" (UniqueName: \"kubernetes.io/projected/a77be5ae-4aa7-4b53-a6c2-4675749db923-kube-api-access-ptsmh\") pod \"payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln\" (UID: \"a77be5ae-4aa7-4b53-a6c2-4675749db923\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:56:58.846082 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.846052 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx"] Sep 10 15:56:58.846295 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.846250 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" podUID="9142424f-1159-4934-9f39-c59cebcfb54c" containerName="payload-processing" containerID="cri-o://26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97" gracePeriod=30 Sep 10 15:56:58.853819 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.853791 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94"] Sep 10 15:56:58.854410 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.854336 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" podUID="562e010c-8db8-4a06-89c1-cfc6de11277b" containerName="payload-pre-processing" containerID="cri-o://9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910" gracePeriod=30 Sep 10 15:56:58.917592 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:58.917564 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:56:59.051669 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.051635 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62"] Sep 10 15:56:59.055097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.055073 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-761990-cdn62"] Sep 10 15:56:59.148845 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.148826 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:56:59.151413 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.151379 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:56:59.203292 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:56:59.203260 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77be5ae_4aa7_4b53_a6c2_4675749db923.slice/crio-a2f1770a9413819bf15d06de7652be1811f58c9f033b969ca793a95750c0df71 WatchSource:0}: Error finding container a2f1770a9413819bf15d06de7652be1811f58c9f033b969ca793a95750c0df71: Status 404 returned error can't find the container with id a2f1770a9413819bf15d06de7652be1811f58c9f033b969ca793a95750c0df71 Sep 10 15:56:59.203805 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.203778 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln"] Sep 10 15:56:59.225066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.222208 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142424f-1159-4934-9f39-c59cebcfb54c-plugins-config-volume\") pod \"9142424f-1159-4934-9f39-c59cebcfb54c\" (UID: \"9142424f-1159-4934-9f39-c59cebcfb54c\") " Sep 10 15:56:59.225066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.222284 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgjw8\" (UniqueName: \"kubernetes.io/projected/562e010c-8db8-4a06-89c1-cfc6de11277b-kube-api-access-tgjw8\") pod \"562e010c-8db8-4a06-89c1-cfc6de11277b\" (UID: \"562e010c-8db8-4a06-89c1-cfc6de11277b\") " Sep 10 15:56:59.225066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.222319 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562e010c-8db8-4a06-89c1-cfc6de11277b-plugins-config-volume\") pod \"562e010c-8db8-4a06-89c1-cfc6de11277b\" (UID: \"562e010c-8db8-4a06-89c1-cfc6de11277b\") " Sep 10 15:56:59.225066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.222365 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqqjw\" (UniqueName: \"kubernetes.io/projected/9142424f-1159-4934-9f39-c59cebcfb54c-kube-api-access-zqqjw\") pod \"9142424f-1159-4934-9f39-c59cebcfb54c\" (UID: \"9142424f-1159-4934-9f39-c59cebcfb54c\") " Sep 10 15:56:59.225066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.223457 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9142424f-1159-4934-9f39-c59cebcfb54c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9142424f-1159-4934-9f39-c59cebcfb54c" (UID: "9142424f-1159-4934-9f39-c59cebcfb54c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:59.225066 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.223731 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562e010c-8db8-4a06-89c1-cfc6de11277b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "562e010c-8db8-4a06-89c1-cfc6de11277b" (UID: "562e010c-8db8-4a06-89c1-cfc6de11277b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:59.228001 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.227907 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9142424f-1159-4934-9f39-c59cebcfb54c-kube-api-access-zqqjw" (OuterVolumeSpecName: "kube-api-access-zqqjw") pod "9142424f-1159-4934-9f39-c59cebcfb54c" (UID: "9142424f-1159-4934-9f39-c59cebcfb54c"). InnerVolumeSpecName "kube-api-access-zqqjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:59.230141 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.230116 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562e010c-8db8-4a06-89c1-cfc6de11277b-kube-api-access-tgjw8" (OuterVolumeSpecName: "kube-api-access-tgjw8") pod "562e010c-8db8-4a06-89c1-cfc6de11277b" (UID: "562e010c-8db8-4a06-89c1-cfc6de11277b"). InnerVolumeSpecName "kube-api-access-tgjw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:59.323273 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.323237 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgjw8\" (UniqueName: \"kubernetes.io/projected/562e010c-8db8-4a06-89c1-cfc6de11277b-kube-api-access-tgjw8\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:59.323273 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.323265 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562e010c-8db8-4a06-89c1-cfc6de11277b-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:59.323477 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.323280 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqqjw\" (UniqueName: \"kubernetes.io/projected/9142424f-1159-4934-9f39-c59cebcfb54c-kube-api-access-zqqjw\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:59.323477 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.323293 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9142424f-1159-4934-9f39-c59cebcfb54c-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:56:59.909324 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.909287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" event={"ID":"a77be5ae-4aa7-4b53-a6c2-4675749db923","Type":"ContainerStarted","Data":"4656da93adb42c7f4085991deb2f0030ddef3d49d97ff0905fc1f1d1962cc2e4"} Sep 10 15:56:59.909899 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.909352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" event={"ID":"a77be5ae-4aa7-4b53-a6c2-4675749db923","Type":"ContainerStarted","Data":"a2f1770a9413819bf15d06de7652be1811f58c9f033b969ca793a95750c0df71"} Sep 10 15:56:59.909899 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.909521 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:56:59.910692 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.910667 2580 generic.go:356] "Generic (PLEG): container finished" podID="9142424f-1159-4934-9f39-c59cebcfb54c" containerID="26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97" exitCode=0 Sep 10 15:56:59.910795 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.910703 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" Sep 10 15:56:59.910795 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.910752 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" event={"ID":"9142424f-1159-4934-9f39-c59cebcfb54c","Type":"ContainerDied","Data":"26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97"} Sep 10 15:56:59.910918 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.910805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx" event={"ID":"9142424f-1159-4934-9f39-c59cebcfb54c","Type":"ContainerDied","Data":"a08bfefc7b6abde79952c55e20035a143f8293477f5669533c672b85994b6b40"} Sep 10 15:56:59.910918 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.910837 2580 scope.go:117] "RemoveContainer" containerID="26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97" Sep 10 15:56:59.912682 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.912661 2580 generic.go:356] "Generic (PLEG): container finished" podID="562e010c-8db8-4a06-89c1-cfc6de11277b" containerID="9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910" exitCode=0 Sep 10 15:56:59.912820 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.912736 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" Sep 10 15:56:59.912820 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.912773 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" event={"ID":"562e010c-8db8-4a06-89c1-cfc6de11277b","Type":"ContainerDied","Data":"9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910"} Sep 10 15:56:59.912820 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.912812 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94" event={"ID":"562e010c-8db8-4a06-89c1-cfc6de11277b","Type":"ContainerDied","Data":"b54a2035064f362028e42f1026e9c511306c641d9dca7b811e3a9c6fc088b852"} Sep 10 15:56:59.922278 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.922258 2580 scope.go:117] "RemoveContainer" containerID="26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97" Sep 10 15:56:59.922584 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:56:59.922556 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97\": container with ID starting with 26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97 not found: ID does not exist" containerID="26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97" Sep 10 15:56:59.922676 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.922593 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97"} err="failed to get container status \"26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97\": rpc error: code = NotFound desc = could not find container \"26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97\": container with ID starting with 26aa8d6a4dc272fd85cd849f3dc3bb85e58ec5927605895ec156db828e4d9a97 not found: ID does not exist" Sep 10 15:56:59.922676 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.922615 2580 scope.go:117] "RemoveContainer" containerID="9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910" Sep 10 15:56:59.927597 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.927555 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" podStartSLOduration=1.927540725 podStartE2EDuration="1.927540725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:59.925205907 +0000 UTC m=+1478.487515098" watchObservedRunningTime="2026-09-10 15:56:59.927540725 +0000 UTC m=+1478.489849915" Sep 10 15:56:59.932973 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.932954 2580 scope.go:117] "RemoveContainer" containerID="9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910" Sep 10 15:56:59.933262 ip-10-0-134-9 kubenswrapper[2580]: E0910 15:56:59.933240 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910\": container with ID starting with 9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910 not found: ID does not exist" containerID="9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910" Sep 10 15:56:59.933345 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.933267 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910"} err="failed to get container status \"9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910\": rpc error: code = NotFound desc = could not find container \"9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910\": container with ID starting with 9f59a9f3df14071d6e2e7979fcb95d1a42100a620711eda7cf90252ad6422910 not found: ID does not exist" Sep 10 15:56:59.938567 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.938541 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx"] Sep 10 15:56:59.941908 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.941887 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-761990-5884cdd755-92rxx"] Sep 10 15:56:59.951788 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.951762 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94"] Sep 10 15:56:59.954964 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:56:59.954939 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-761990-579678678b-s8g94"] Sep 10 15:57:00.029694 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:00.029608 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="562e010c-8db8-4a06-89c1-cfc6de11277b" path="/var/lib/kubelet/pods/562e010c-8db8-4a06-89c1-cfc6de11277b/volumes" Sep 10 15:57:00.030081 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:00.030057 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9142424f-1159-4934-9f39-c59cebcfb54c" path="/var/lib/kubelet/pods/9142424f-1159-4934-9f39-c59cebcfb54c/volumes" Sep 10 15:57:00.030439 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:00.030404 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03cdabd-239c-4eb9-9b38-aa1e0f479d33" path="/var/lib/kubelet/pods/c03cdabd-239c-4eb9-9b38-aa1e0f479d33/volumes" Sep 10 15:57:10.922723 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:10.922691 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:57:11.924649 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:11.924602 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 15:57:11.925120 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:11.924687 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 15:57:12.988660 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:12.988624 2580 generic.go:356] "Generic (PLEG): container finished" podID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerID="2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d" exitCode=0 Sep 10 15:57:12.989196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:12.988673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" event={"ID":"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b","Type":"ContainerDied","Data":"2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d"} Sep 10 15:57:12.989196 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:12.989186 2580 scope.go:117] "RemoveContainer" containerID="2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d" Sep 10 15:57:12.990290 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:12.990263 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:57:12.990400 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:12.990329 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:57:12.990400 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:12.990362 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"} Sep 10 15:57:13.996536 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:13.996498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" event={"ID":"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b","Type":"ContainerStarted","Data":"e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251"} Sep 10 15:57:14.484566 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:14.484528 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:57:14.484815 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:14.484792 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 10 15:57:14.484874 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:14.484855 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 10 15:57:15.484965 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:15.484925 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 10 15:57:15.485357 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:15.484994 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 10 15:57:16.485277 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:16.485242 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 10 15:57:16.485705 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:16.485296 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 10 15:57:17.485550 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:17.485507 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 10 15:57:17.485943 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:17.485583 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 10 15:57:18.485068 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:18.485031 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 10 15:57:18.485250 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:18.485091 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 10 15:57:19.485143 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:19.485103 2580 patch_prober.go:28] interesting pod/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Sep 10 15:57:19.485652 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:19.485161 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Sep 10 15:57:20.488676 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:20.488643 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:57:20.489056 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:20.488953 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:57:20.489569 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:20.489554 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 15:57:22.070564 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:22.070537 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:57:22.073529 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:22.073507 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 15:57:27.464839 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.464805 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd"] Sep 10 15:57:27.465471 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.465450 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9142424f-1159-4934-9f39-c59cebcfb54c" containerName="payload-processing" Sep 10 15:57:27.465565 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.465474 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9142424f-1159-4934-9f39-c59cebcfb54c" containerName="payload-processing" Sep 10 15:57:27.465565 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.465492 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="562e010c-8db8-4a06-89c1-cfc6de11277b" containerName="payload-pre-processing" Sep 10 15:57:27.465565 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.465500 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="562e010c-8db8-4a06-89c1-cfc6de11277b" containerName="payload-pre-processing" Sep 10 15:57:27.465698 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.465586 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="562e010c-8db8-4a06-89c1-cfc6de11277b" containerName="payload-pre-processing" Sep 10 15:57:27.465698 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.465601 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9142424f-1159-4934-9f39-c59cebcfb54c" containerName="payload-processing" Sep 10 15:57:27.468676 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.468659 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:27.471021 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.471001 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f52559\"" Sep 10 15:57:27.471124 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.471005 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f52559-dockercfg-x9j29\"" Sep 10 15:57:27.477880 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.477860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd"] Sep 10 15:57:27.500848 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.500818 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78kfb\" (UniqueName: \"kubernetes.io/projected/e16904bf-0be6-4963-8ceb-c06342915cd3-kube-api-access-78kfb\") pod \"payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd\" (UID: \"e16904bf-0be6-4963-8ceb-c06342915cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:27.500961 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.500929 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16904bf-0be6-4963-8ceb-c06342915cd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd\" (UID: \"e16904bf-0be6-4963-8ceb-c06342915cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:27.503990 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.503970 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w"] Sep 10 15:57:27.506636 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.506621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:27.518970 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.518950 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w"] Sep 10 15:57:27.601838 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.601803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78kfb\" (UniqueName: \"kubernetes.io/projected/e16904bf-0be6-4963-8ceb-c06342915cd3-kube-api-access-78kfb\") pod \"payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd\" (UID: \"e16904bf-0be6-4963-8ceb-c06342915cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:27.602049 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.601885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc47s\" (UniqueName: \"kubernetes.io/projected/772663fc-7928-467a-bc53-8d87e0ce6c68-kube-api-access-cc47s\") pod \"payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w\" (UID: \"772663fc-7928-467a-bc53-8d87e0ce6c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:27.602049 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.602014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16904bf-0be6-4963-8ceb-c06342915cd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd\" (UID: \"e16904bf-0be6-4963-8ceb-c06342915cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:27.602179 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.602072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772663fc-7928-467a-bc53-8d87e0ce6c68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w\" (UID: \"772663fc-7928-467a-bc53-8d87e0ce6c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:27.603198 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.603172 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16904bf-0be6-4963-8ceb-c06342915cd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd\" (UID: \"e16904bf-0be6-4963-8ceb-c06342915cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:27.610209 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.610179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78kfb\" (UniqueName: \"kubernetes.io/projected/e16904bf-0be6-4963-8ceb-c06342915cd3-kube-api-access-78kfb\") pod \"payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd\" (UID: \"e16904bf-0be6-4963-8ceb-c06342915cd3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:27.702942 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.702908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772663fc-7928-467a-bc53-8d87e0ce6c68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w\" (UID: \"772663fc-7928-467a-bc53-8d87e0ce6c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:27.703109 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.702978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc47s\" (UniqueName: \"kubernetes.io/projected/772663fc-7928-467a-bc53-8d87e0ce6c68-kube-api-access-cc47s\") pod \"payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w\" (UID: \"772663fc-7928-467a-bc53-8d87e0ce6c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:27.703579 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.703560 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772663fc-7928-467a-bc53-8d87e0ce6c68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w\" (UID: \"772663fc-7928-467a-bc53-8d87e0ce6c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:27.710540 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.710516 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc47s\" (UniqueName: \"kubernetes.io/projected/772663fc-7928-467a-bc53-8d87e0ce6c68-kube-api-access-cc47s\") pod \"payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w\" (UID: \"772663fc-7928-467a-bc53-8d87e0ce6c68\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:27.780078 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.780007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:27.815953 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:27.815927 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:28.058579 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:28.058548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd"] Sep 10 15:57:28.061328 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:57:28.061300 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16904bf_0be6_4963_8ceb_c06342915cd3.slice/crio-0b6c22fb5f9c26ce239cbbe8bac47f53f956a67c1285d42d130acc1c14997d81 WatchSource:0}: Error finding container 0b6c22fb5f9c26ce239cbbe8bac47f53f956a67c1285d42d130acc1c14997d81: Status 404 returned error can't find the container with id 0b6c22fb5f9c26ce239cbbe8bac47f53f956a67c1285d42d130acc1c14997d81 Sep 10 15:57:28.094673 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:28.094641 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" event={"ID":"e16904bf-0be6-4963-8ceb-c06342915cd3","Type":"ContainerStarted","Data":"0b6c22fb5f9c26ce239cbbe8bac47f53f956a67c1285d42d130acc1c14997d81"} Sep 10 15:57:28.102372 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:28.102345 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w"] Sep 10 15:57:28.103813 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:57:28.103791 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772663fc_7928_467a_bc53_8d87e0ce6c68.slice/crio-a2a75b38f1fbbc4133119c8a215d4962b20fdfc5c4c2474c15788ce0342004fa WatchSource:0}: Error finding container a2a75b38f1fbbc4133119c8a215d4962b20fdfc5c4c2474c15788ce0342004fa: Status 404 returned error can't find the container with id a2a75b38f1fbbc4133119c8a215d4962b20fdfc5c4c2474c15788ce0342004fa Sep 10 15:57:29.102624 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:29.102583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" event={"ID":"e16904bf-0be6-4963-8ceb-c06342915cd3","Type":"ContainerStarted","Data":"b909905ed3a4310b17d1f8c01ac896a1cabc7c432adfa3a2fdaf6e8f11aff98e"} Sep 10 15:57:29.103051 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:29.102650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:29.104039 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:29.104017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" event={"ID":"772663fc-7928-467a-bc53-8d87e0ce6c68","Type":"ContainerStarted","Data":"65da0d5fa4c83b45fadeb49ffd48924880abc740fb73c609b37eb813f74a4030"} Sep 10 15:57:29.104097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:29.104067 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:29.104097 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:29.104083 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" event={"ID":"772663fc-7928-467a-bc53-8d87e0ce6c68","Type":"ContainerStarted","Data":"a2a75b38f1fbbc4133119c8a215d4962b20fdfc5c4c2474c15788ce0342004fa"} Sep 10 15:57:29.118707 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:29.118658 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" podStartSLOduration=2.118646164 podStartE2EDuration="2.118646164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:29.11677795 +0000 UTC m=+1507.679087142" watchObservedRunningTime="2026-09-10 15:57:29.118646164 +0000 UTC m=+1507.680955354" Sep 10 15:57:29.132918 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:29.132871 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" podStartSLOduration=2.132858652 podStartE2EDuration="2.132858652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:29.131320452 +0000 UTC m=+1507.693629643" watchObservedRunningTime="2026-09-10 15:57:29.132858652 +0000 UTC m=+1507.695167842" Sep 10 15:57:40.111630 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:40.111590 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 15:57:40.112045 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:40.111919 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 15:57:55.768953 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:55.768882 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qb9kv_fe46e4b2-e73b-4192-a133-bc065eb7f948/payload-processing/0.log" Sep 10 15:57:55.889789 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:57:55.889740 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w_772663fc-7928-467a-bc53-8d87e0ce6c68/payload-processing/0.log" Sep 10 15:58:36.677131 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:58:36.677075 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln_a77be5ae-4aa7-4b53-a6c2-4675749db923/payload-processing/0.log" Sep 10 15:58:36.804931 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:58:36.804895 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qb9kv_fe46e4b2-e73b-4192-a133-bc065eb7f948/payload-processing/0.log" Sep 10 15:58:36.936347 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:58:36.936267 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w_772663fc-7928-467a-bc53-8d87e0ce6c68/payload-processing/0.log" Sep 10 15:59:06.739135 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:06.739098 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6"] Sep 10 15:59:06.741770 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:06.741753 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" Sep 10 15:59:06.744021 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:06.743999 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:59:06.744156 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:06.744032 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:59:06.749375 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:06.749344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6"] Sep 10 15:59:06.866619 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:06.866582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991b8275-70dc-4d72-82cc-fd14df47a5bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6\" (UID: \"991b8275-70dc-4d72-82cc-fd14df47a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" Sep 10 15:59:06.967852 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:06.967817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991b8275-70dc-4d72-82cc-fd14df47a5bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6\" (UID: \"991b8275-70dc-4d72-82cc-fd14df47a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" Sep 10 15:59:06.968489 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:06.968470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991b8275-70dc-4d72-82cc-fd14df47a5bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6\" (UID: \"991b8275-70dc-4d72-82cc-fd14df47a5bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" Sep 10 15:59:07.054693 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:07.054596 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" Sep 10 15:59:07.330633 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:07.330609 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6"] Sep 10 15:59:07.332768 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:59:07.332735 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991b8275_70dc_4d72_82cc_fd14df47a5bf.slice/crio-06df1636db5ae2a3abd9e5bad19976f6d72ed1a06d1514d99909f9a3d9f5c942 WatchSource:0}: Error finding container 06df1636db5ae2a3abd9e5bad19976f6d72ed1a06d1514d99909f9a3d9f5c942: Status 404 returned error can't find the container with id 06df1636db5ae2a3abd9e5bad19976f6d72ed1a06d1514d99909f9a3d9f5c942 Sep 10 15:59:07.622488 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:07.622462 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" event={"ID":"991b8275-70dc-4d72-82cc-fd14df47a5bf","Type":"ContainerStarted","Data":"06df1636db5ae2a3abd9e5bad19976f6d72ed1a06d1514d99909f9a3d9f5c942"} Sep 10 15:59:08.636618 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:08.636581 2580 generic.go:356] "Generic (PLEG): container finished" podID="991b8275-70dc-4d72-82cc-fd14df47a5bf" containerID="d8e92de448c4fa58fab49bfcf9aa47e83dcd93ad9884d81b2545a302c1ffbe6b" exitCode=0 Sep 10 15:59:08.637038 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:08.636677 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" event={"ID":"991b8275-70dc-4d72-82cc-fd14df47a5bf","Type":"ContainerDied","Data":"d8e92de448c4fa58fab49bfcf9aa47e83dcd93ad9884d81b2545a302c1ffbe6b"} Sep 10 15:59:09.776949 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:09.776923 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" Sep 10 15:59:09.896548 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:09.896460 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991b8275-70dc-4d72-82cc-fd14df47a5bf-maas-api-service-ca\") pod \"991b8275-70dc-4d72-82cc-fd14df47a5bf\" (UID: \"991b8275-70dc-4d72-82cc-fd14df47a5bf\") " Sep 10 15:59:09.896839 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:09.896815 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991b8275-70dc-4d72-82cc-fd14df47a5bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "991b8275-70dc-4d72-82cc-fd14df47a5bf" (UID: "991b8275-70dc-4d72-82cc-fd14df47a5bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:09.997819 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:09.997787 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991b8275-70dc-4d72-82cc-fd14df47a5bf-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:59:10.650931 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:10.650906 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" Sep 10 15:59:10.651107 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:10.650950 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6" event={"ID":"991b8275-70dc-4d72-82cc-fd14df47a5bf","Type":"ContainerDied","Data":"06df1636db5ae2a3abd9e5bad19976f6d72ed1a06d1514d99909f9a3d9f5c942"} Sep 10 15:59:10.651107 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:10.650991 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06df1636db5ae2a3abd9e5bad19976f6d72ed1a06d1514d99909f9a3d9f5c942" Sep 10 15:59:13.076722 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:13.076688 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6"] Sep 10 15:59:13.081557 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:13.081529 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7388f5-9kkj6"] Sep 10 15:59:14.029111 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:14.029031 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="991b8275-70dc-4d72-82cc-fd14df47a5bf" path="/var/lib/kubelet/pods/991b8275-70dc-4d72-82cc-fd14df47a5bf/volumes" Sep 10 15:59:22.360401 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:22.360366 2580 scope.go:117] "RemoveContainer" containerID="e1cc6f6d08f666b14cf52353fd2a05f2b9d32768de21a155ca60c09f8342ed84" Sep 10 15:59:23.097729 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.097694 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf"] Sep 10 15:59:23.098186 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.098168 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="991b8275-70dc-4d72-82cc-fd14df47a5bf" containerName="revoke-keys" Sep 10 15:59:23.098279 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.098188 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="991b8275-70dc-4d72-82cc-fd14df47a5bf" containerName="revoke-keys" Sep 10 15:59:23.098339 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.098305 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="991b8275-70dc-4d72-82cc-fd14df47a5bf" containerName="revoke-keys" Sep 10 15:59:23.100400 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.100379 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" Sep 10 15:59:23.102707 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.102685 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:59:23.102802 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.102715 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 15:59:23.109124 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.108311 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf"] Sep 10 15:59:23.130541 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.130506 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ef26f-9929-43a0-9cf2-26fa67f05c69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf\" (UID: \"5d3ef26f-9929-43a0-9cf2-26fa67f05c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" Sep 10 15:59:23.231693 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.231656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ef26f-9929-43a0-9cf2-26fa67f05c69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf\" (UID: \"5d3ef26f-9929-43a0-9cf2-26fa67f05c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" Sep 10 15:59:23.232268 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.232248 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ef26f-9929-43a0-9cf2-26fa67f05c69-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf\" (UID: \"5d3ef26f-9929-43a0-9cf2-26fa67f05c69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" Sep 10 15:59:23.412373 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.412341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" Sep 10 15:59:23.681927 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.681900 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf"] Sep 10 15:59:23.684133 ip-10-0-134-9 kubenswrapper[2580]: W0910 15:59:23.684099 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3ef26f_9929_43a0_9cf2_26fa67f05c69.slice/crio-4905d2a15e7b914804605b890c3e1ef453c233e485af1bada4f70469da76064e WatchSource:0}: Error finding container 4905d2a15e7b914804605b890c3e1ef453c233e485af1bada4f70469da76064e: Status 404 returned error can't find the container with id 4905d2a15e7b914804605b890c3e1ef453c233e485af1bada4f70469da76064e Sep 10 15:59:23.731444 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:23.731392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" event={"ID":"5d3ef26f-9929-43a0-9cf2-26fa67f05c69","Type":"ContainerStarted","Data":"4905d2a15e7b914804605b890c3e1ef453c233e485af1bada4f70469da76064e"} Sep 10 15:59:24.738208 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:24.738171 2580 generic.go:356] "Generic (PLEG): container finished" podID="5d3ef26f-9929-43a0-9cf2-26fa67f05c69" containerID="d4563042551f803783a1c04e0cbb527b1f716d1c7bbaa8d3a37d8127f7363379" exitCode=0 Sep 10 15:59:24.738619 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:24.738262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" event={"ID":"5d3ef26f-9929-43a0-9cf2-26fa67f05c69","Type":"ContainerDied","Data":"d4563042551f803783a1c04e0cbb527b1f716d1c7bbaa8d3a37d8127f7363379"} Sep 10 15:59:25.883163 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:25.883138 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" Sep 10 15:59:25.958260 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:25.958221 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ef26f-9929-43a0-9cf2-26fa67f05c69-maas-api-service-ca\") pod \"5d3ef26f-9929-43a0-9cf2-26fa67f05c69\" (UID: \"5d3ef26f-9929-43a0-9cf2-26fa67f05c69\") " Sep 10 15:59:25.958592 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:25.958567 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3ef26f-9929-43a0-9cf2-26fa67f05c69-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d3ef26f-9929-43a0-9cf2-26fa67f05c69" (UID: "5d3ef26f-9929-43a0-9cf2-26fa67f05c69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:25.958682 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:25.958652 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ef26f-9929-43a0-9cf2-26fa67f05c69-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:59:26.749867 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:26.749838 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" Sep 10 15:59:26.749867 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:26.749852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf" event={"ID":"5d3ef26f-9929-43a0-9cf2-26fa67f05c69","Type":"ContainerDied","Data":"4905d2a15e7b914804605b890c3e1ef453c233e485af1bada4f70469da76064e"} Sep 10 15:59:26.750089 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:26.749895 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4905d2a15e7b914804605b890c3e1ef453c233e485af1bada4f70469da76064e" Sep 10 15:59:38.591479 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.591441 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln"] Sep 10 15:59:38.592072 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.591689 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" podUID="a77be5ae-4aa7-4b53-a6c2-4675749db923" containerName="payload-processing" containerID="cri-o://4656da93adb42c7f4085991deb2f0030ddef3d49d97ff0905fc1f1d1962cc2e4" gracePeriod=30 Sep 10 15:59:38.795378 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.795340 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf"] Sep 10 15:59:38.798270 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.798247 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a85a91-8ttqf"] Sep 10 15:59:38.819591 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.819560 2580 generic.go:356] "Generic (PLEG): container finished" podID="a77be5ae-4aa7-4b53-a6c2-4675749db923" containerID="4656da93adb42c7f4085991deb2f0030ddef3d49d97ff0905fc1f1d1962cc2e4" exitCode=0 Sep 10 15:59:38.819728 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.819633 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" event={"ID":"a77be5ae-4aa7-4b53-a6c2-4675749db923","Type":"ContainerDied","Data":"4656da93adb42c7f4085991deb2f0030ddef3d49d97ff0905fc1f1d1962cc2e4"} Sep 10 15:59:38.853509 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.853450 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:59:38.995088 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.995056 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptsmh\" (UniqueName: \"kubernetes.io/projected/a77be5ae-4aa7-4b53-a6c2-4675749db923-kube-api-access-ptsmh\") pod \"a77be5ae-4aa7-4b53-a6c2-4675749db923\" (UID: \"a77be5ae-4aa7-4b53-a6c2-4675749db923\") " Sep 10 15:59:38.995280 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.995124 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77be5ae-4aa7-4b53-a6c2-4675749db923-plugins-config-volume\") pod \"a77be5ae-4aa7-4b53-a6c2-4675749db923\" (UID: \"a77be5ae-4aa7-4b53-a6c2-4675749db923\") " Sep 10 15:59:38.995578 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.995554 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77be5ae-4aa7-4b53-a6c2-4675749db923-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a77be5ae-4aa7-4b53-a6c2-4675749db923" (UID: "a77be5ae-4aa7-4b53-a6c2-4675749db923"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:38.997203 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:38.997183 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77be5ae-4aa7-4b53-a6c2-4675749db923-kube-api-access-ptsmh" (OuterVolumeSpecName: "kube-api-access-ptsmh") pod "a77be5ae-4aa7-4b53-a6c2-4675749db923" (UID: "a77be5ae-4aa7-4b53-a6c2-4675749db923"). InnerVolumeSpecName "kube-api-access-ptsmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:39.096362 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:39.096322 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptsmh\" (UniqueName: \"kubernetes.io/projected/a77be5ae-4aa7-4b53-a6c2-4675749db923-kube-api-access-ptsmh\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:59:39.096362 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:39.096352 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77be5ae-4aa7-4b53-a6c2-4675749db923-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 15:59:39.825980 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:39.825949 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" Sep 10 15:59:39.826359 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:39.825979 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln" event={"ID":"a77be5ae-4aa7-4b53-a6c2-4675749db923","Type":"ContainerDied","Data":"a2f1770a9413819bf15d06de7652be1811f58c9f033b969ca793a95750c0df71"} Sep 10 15:59:39.826359 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:39.826041 2580 scope.go:117] "RemoveContainer" containerID="4656da93adb42c7f4085991deb2f0030ddef3d49d97ff0905fc1f1d1962cc2e4" Sep 10 15:59:39.847736 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:39.847715 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln"] Sep 10 15:59:39.851939 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:39.851917 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a85a91-7b8674bd7f-zsdln"] Sep 10 15:59:40.028473 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:40.028403 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d3ef26f-9929-43a0-9cf2-26fa67f05c69" path="/var/lib/kubelet/pods/5d3ef26f-9929-43a0-9cf2-26fa67f05c69/volumes" Sep 10 15:59:40.028871 ip-10-0-134-9 kubenswrapper[2580]: I0910 15:59:40.028856 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77be5ae-4aa7-4b53-a6c2-4675749db923" path="/var/lib/kubelet/pods/a77be5ae-4aa7-4b53-a6c2-4675749db923/volumes" Sep 10 16:00:00.140223 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.140187 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7"] Sep 10 16:00:00.140785 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.140699 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3ef26f-9929-43a0-9cf2-26fa67f05c69" containerName="revoke-keys" Sep 10 16:00:00.140785 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.140720 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3ef26f-9929-43a0-9cf2-26fa67f05c69" containerName="revoke-keys" Sep 10 16:00:00.140785 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.140764 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77be5ae-4aa7-4b53-a6c2-4675749db923" containerName="payload-processing" Sep 10 16:00:00.140785 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.140772 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77be5ae-4aa7-4b53-a6c2-4675749db923" containerName="payload-processing" Sep 10 16:00:00.141046 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.140875 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3ef26f-9929-43a0-9cf2-26fa67f05c69" containerName="revoke-keys" Sep 10 16:00:00.141046 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.140891 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77be5ae-4aa7-4b53-a6c2-4675749db923" containerName="payload-processing" Sep 10 16:00:00.143450 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.143432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7" Sep 10 16:00:00.143929 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.143908 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn"] Sep 10 16:00:00.145864 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.145843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 16:00:00.146504 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.146487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn" Sep 10 16:00:00.152030 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.152006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7"] Sep 10 16:00:00.154648 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.154624 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn"] Sep 10 16:00:00.159092 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.159074 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7" Sep 10 16:00:00.167617 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.167595 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn" Sep 10 16:00:00.439201 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.439177 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7"] Sep 10 16:00:00.441059 ip-10-0-134-9 kubenswrapper[2580]: W0910 16:00:00.441030 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31c9fd0_7c60_4391_a5dc_a7398af96780.slice/crio-33ad29b1e6b012f179870411687d7e08952f689e550bf72727b71cceb084620a WatchSource:0}: Error finding container 33ad29b1e6b012f179870411687d7e08952f689e550bf72727b71cceb084620a: Status 404 returned error can't find the container with id 33ad29b1e6b012f179870411687d7e08952f689e550bf72727b71cceb084620a Sep 10 16:00:00.446076 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.445855 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn"] Sep 10 16:00:00.447837 ip-10-0-134-9 kubenswrapper[2580]: W0910 16:00:00.447811 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e717b1_731a_4d65_8e02_32d95b23b00d.slice/crio-6df9f5eea6e21cf31f34a2391b119ad7a400405287268eaeb809f525afd59348 WatchSource:0}: Error finding container 6df9f5eea6e21cf31f34a2391b119ad7a400405287268eaeb809f525afd59348: Status 404 returned error can't find the container with id 6df9f5eea6e21cf31f34a2391b119ad7a400405287268eaeb809f525afd59348 Sep 10 16:00:00.940123 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.940074 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7" event={"ID":"c31c9fd0-7c60-4391-a5dc-a7398af96780","Type":"ContainerStarted","Data":"eb645a72da420d39506ad721c75581a0f4c470b36e0f6360388a28ab4c540d43"} Sep 10 16:00:00.940330 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.940188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7" event={"ID":"c31c9fd0-7c60-4391-a5dc-a7398af96780","Type":"ContainerStarted","Data":"33ad29b1e6b012f179870411687d7e08952f689e550bf72727b71cceb084620a"} Sep 10 16:00:00.941646 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.941621 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn" event={"ID":"31e717b1-731a-4d65-8e02-32d95b23b00d","Type":"ContainerStarted","Data":"c637414233e23fa0f81a83a391cc75bdcfba9bc3f97ddc167090a6db25930f68"} Sep 10 16:00:00.941737 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.941667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn" event={"ID":"31e717b1-731a-4d65-8e02-32d95b23b00d","Type":"ContainerStarted","Data":"6df9f5eea6e21cf31f34a2391b119ad7a400405287268eaeb809f525afd59348"} Sep 10 16:00:00.956143 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.956101 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7" podStartSLOduration=0.956088278 podStartE2EDuration="956.088278ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:00.954517756 +0000 UTC m=+1659.516826948" watchObservedRunningTime="2026-09-10 16:00:00.956088278 +0000 UTC m=+1659.518397467" Sep 10 16:00:00.968602 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:00.968559 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn" podStartSLOduration=0.968545838 podStartE2EDuration="968.545838ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:00.966682128 +0000 UTC m=+1659.528991321" watchObservedRunningTime="2026-09-10 16:00:00.968545838 +0000 UTC m=+1659.530855028" Sep 10 16:00:01.949374 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:01.949337 2580 generic.go:356] "Generic (PLEG): container finished" podID="31e717b1-731a-4d65-8e02-32d95b23b00d" containerID="c637414233e23fa0f81a83a391cc75bdcfba9bc3f97ddc167090a6db25930f68" exitCode=0 Sep 10 16:00:01.949849 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:01.949446 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn" event={"ID":"31e717b1-731a-4d65-8e02-32d95b23b00d","Type":"ContainerDied","Data":"c637414233e23fa0f81a83a391cc75bdcfba9bc3f97ddc167090a6db25930f68"} Sep 10 16:00:01.950847 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:01.950825 2580 generic.go:356] "Generic (PLEG): container finished" podID="c31c9fd0-7c60-4391-a5dc-a7398af96780" containerID="eb645a72da420d39506ad721c75581a0f4c470b36e0f6360388a28ab4c540d43" exitCode=0 Sep 10 16:00:01.950964 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:01.950869 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7" event={"ID":"c31c9fd0-7c60-4391-a5dc-a7398af96780","Type":"ContainerDied","Data":"eb645a72da420d39506ad721c75581a0f4c470b36e0f6360388a28ab4c540d43"} Sep 10 16:00:03.085182 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:03.085156 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn" Sep 10 16:00:03.115150 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:03.115128 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7" Sep 10 16:00:03.962593 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:03.962564 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn" Sep 10 16:00:03.962778 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:03.962616 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn" event={"ID":"31e717b1-731a-4d65-8e02-32d95b23b00d","Type":"ContainerDied","Data":"6df9f5eea6e21cf31f34a2391b119ad7a400405287268eaeb809f525afd59348"} Sep 10 16:00:03.962778 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:03.962663 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df9f5eea6e21cf31f34a2391b119ad7a400405287268eaeb809f525afd59348" Sep 10 16:00:03.963839 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:03.963821 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7" Sep 10 16:00:03.963945 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:03.963863 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7" event={"ID":"c31c9fd0-7c60-4391-a5dc-a7398af96780","Type":"ContainerDied","Data":"33ad29b1e6b012f179870411687d7e08952f689e550bf72727b71cceb084620a"} Sep 10 16:00:03.963945 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:03.963898 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33ad29b1e6b012f179870411687d7e08952f689e550bf72727b71cceb084620a" Sep 10 16:00:27.066026 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:27.065995 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s"] Sep 10 16:00:27.066502 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:27.066247 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" containerID="cri-o://e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251" gracePeriod=30 Sep 10 16:00:32.327996 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.327938 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 16:00:32.413578 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413550 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-envoy\") pod \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " Sep 10 16:00:32.413578 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413581 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-credential-socket\") pod \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " Sep 10 16:00:32.413782 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413629 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-token\") pod \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " Sep 10 16:00:32.413782 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413660 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-certs\") pod \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " Sep 10 16:00:32.413782 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413681 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-socket\") pod \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " Sep 10 16:00:32.413782 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413720 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istiod-ca-cert\") pod \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " Sep 10 16:00:32.413782 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413770 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-podinfo\") pod \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " Sep 10 16:00:32.414041 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413813 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-data\") pod \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " Sep 10 16:00:32.414041 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413838 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpvll\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-kube-api-access-dpvll\") pod \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\" (UID: \"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b\") " Sep 10 16:00:32.414041 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413961 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" (UID: "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:32.414041 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.413973 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" (UID: "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:32.414264 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.414041 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" (UID: "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:32.414264 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.414193 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" (UID: "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:32.414379 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.414302 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-data\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:32.414379 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.414315 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-credential-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:32.414379 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.414326 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:32.414379 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.414334 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-workload-socket\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:32.414589 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.414395 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" (UID: "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:32.415787 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.415759 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" (UID: "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:32.415787 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.415779 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" (UID: "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:32.416549 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.416531 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" (UID: "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:00:32.416620 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.416546 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-kube-api-access-dpvll" (OuterVolumeSpecName: "kube-api-access-dpvll") pod "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" (UID: "fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b"). InnerVolumeSpecName "kube-api-access-dpvll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:32.514914 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.514876 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-token\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:32.514914 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.514902 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istiod-ca-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:32.514914 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.514914 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-podinfo\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:32.515183 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.514925 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpvll\" (UniqueName: \"kubernetes.io/projected/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-kube-api-access-dpvll\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:32.515183 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:32.514935 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b-istio-envoy\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:33.120356 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.120323 2580 generic.go:356] "Generic (PLEG): container finished" podID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerID="e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251" exitCode=0 Sep 10 16:00:33.120550 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.120380 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" Sep 10 16:00:33.120550 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.120412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" event={"ID":"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b","Type":"ContainerDied","Data":"e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251"} Sep 10 16:00:33.120550 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.120480 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s" event={"ID":"fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b","Type":"ContainerDied","Data":"1f0c6cf041a9747bcfccb7d5d42e8d16011ed086c1f158c5b76fa7f133efcc39"} Sep 10 16:00:33.120550 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.120503 2580 scope.go:117] "RemoveContainer" containerID="e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251" Sep 10 16:00:33.129677 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.129660 2580 scope.go:117] "RemoveContainer" containerID="2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d" Sep 10 16:00:33.141057 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.140969 2580 scope.go:117] "RemoveContainer" containerID="e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251" Sep 10 16:00:33.141444 ip-10-0-134-9 kubenswrapper[2580]: E0910 16:00:33.141406 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251\": container with ID starting with e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251 not found: ID does not exist" containerID="e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251" Sep 10 16:00:33.141546 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.141456 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251"} err="failed to get container status \"e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251\": rpc error: code = NotFound desc = could not find container \"e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251\": container with ID starting with e9ce94c5348943b0f6832490b5e1e4a2bea42be06b19712fe8210ecafb405251 not found: ID does not exist" Sep 10 16:00:33.141546 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.141475 2580 scope.go:117] "RemoveContainer" containerID="2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d" Sep 10 16:00:33.141781 ip-10-0-134-9 kubenswrapper[2580]: E0910 16:00:33.141764 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d\": container with ID starting with 2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d not found: ID does not exist" containerID="2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d" Sep 10 16:00:33.141847 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.141786 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d"} err="failed to get container status \"2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d\": rpc error: code = NotFound desc = could not find container \"2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d\": container with ID starting with 2b4cb1d8b880d2fc20aa2f4ecf286d42be8ef85fa394c3b1b5b5d3f22d17917d not found: ID does not exist" Sep 10 16:00:33.143540 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.143520 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s"] Sep 10 16:00:33.146590 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:33.146571 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a85a91-openshift-default-86d97b7456-bj76s"] Sep 10 16:00:34.028708 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:34.028666 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" path="/var/lib/kubelet/pods/fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b/volumes" Sep 10 16:00:37.910437 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:37.910391 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7"] Sep 10 16:00:37.914750 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:37.914723 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f52559-29817600-z95b7"] Sep 10 16:00:37.937858 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:37.937825 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w"] Sep 10 16:00:37.938113 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:37.938085 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" podUID="772663fc-7928-467a-bc53-8d87e0ce6c68" containerName="payload-processing" containerID="cri-o://65da0d5fa4c83b45fadeb49ffd48924880abc740fb73c609b37eb813f74a4030" gracePeriod=30 Sep 10 16:00:37.943893 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:37.943865 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd"] Sep 10 16:00:37.944402 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:37.944297 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" podUID="e16904bf-0be6-4963-8ceb-c06342915cd3" containerName="payload-pre-processing" containerID="cri-o://b909905ed3a4310b17d1f8c01ac896a1cabc7c432adfa3a2fdaf6e8f11aff98e" gracePeriod=30 Sep 10 16:00:38.028968 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.028937 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31c9fd0-7c60-4391-a5dc-a7398af96780" path="/var/lib/kubelet/pods/c31c9fd0-7c60-4391-a5dc-a7398af96780/volumes" Sep 10 16:00:38.151539 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.151503 2580 generic.go:356] "Generic (PLEG): container finished" podID="e16904bf-0be6-4963-8ceb-c06342915cd3" containerID="b909905ed3a4310b17d1f8c01ac896a1cabc7c432adfa3a2fdaf6e8f11aff98e" exitCode=0 Sep 10 16:00:38.151717 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.151572 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" event={"ID":"e16904bf-0be6-4963-8ceb-c06342915cd3","Type":"ContainerDied","Data":"b909905ed3a4310b17d1f8c01ac896a1cabc7c432adfa3a2fdaf6e8f11aff98e"} Sep 10 16:00:38.154337 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.154296 2580 generic.go:356] "Generic (PLEG): container finished" podID="772663fc-7928-467a-bc53-8d87e0ce6c68" containerID="65da0d5fa4c83b45fadeb49ffd48924880abc740fb73c609b37eb813f74a4030" exitCode=0 Sep 10 16:00:38.154486 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.154341 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" event={"ID":"772663fc-7928-467a-bc53-8d87e0ce6c68","Type":"ContainerDied","Data":"65da0d5fa4c83b45fadeb49ffd48924880abc740fb73c609b37eb813f74a4030"} Sep 10 16:00:38.198820 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.198799 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 16:00:38.229213 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.229189 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 16:00:38.266909 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.266882 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772663fc-7928-467a-bc53-8d87e0ce6c68-plugins-config-volume\") pod \"772663fc-7928-467a-bc53-8d87e0ce6c68\" (UID: \"772663fc-7928-467a-bc53-8d87e0ce6c68\") " Sep 10 16:00:38.267722 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.266933 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16904bf-0be6-4963-8ceb-c06342915cd3-plugins-config-volume\") pod \"e16904bf-0be6-4963-8ceb-c06342915cd3\" (UID: \"e16904bf-0be6-4963-8ceb-c06342915cd3\") " Sep 10 16:00:38.267722 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.267030 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78kfb\" (UniqueName: \"kubernetes.io/projected/e16904bf-0be6-4963-8ceb-c06342915cd3-kube-api-access-78kfb\") pod \"e16904bf-0be6-4963-8ceb-c06342915cd3\" (UID: \"e16904bf-0be6-4963-8ceb-c06342915cd3\") " Sep 10 16:00:38.267722 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.267058 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc47s\" (UniqueName: \"kubernetes.io/projected/772663fc-7928-467a-bc53-8d87e0ce6c68-kube-api-access-cc47s\") pod \"772663fc-7928-467a-bc53-8d87e0ce6c68\" (UID: \"772663fc-7928-467a-bc53-8d87e0ce6c68\") " Sep 10 16:00:38.267722 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.267388 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16904bf-0be6-4963-8ceb-c06342915cd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e16904bf-0be6-4963-8ceb-c06342915cd3" (UID: "e16904bf-0be6-4963-8ceb-c06342915cd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:38.267722 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.267547 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e16904bf-0be6-4963-8ceb-c06342915cd3-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:38.268039 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.267835 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772663fc-7928-467a-bc53-8d87e0ce6c68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "772663fc-7928-467a-bc53-8d87e0ce6c68" (UID: "772663fc-7928-467a-bc53-8d87e0ce6c68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:38.269689 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.269662 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772663fc-7928-467a-bc53-8d87e0ce6c68-kube-api-access-cc47s" (OuterVolumeSpecName: "kube-api-access-cc47s") pod "772663fc-7928-467a-bc53-8d87e0ce6c68" (UID: "772663fc-7928-467a-bc53-8d87e0ce6c68"). InnerVolumeSpecName "kube-api-access-cc47s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:38.270478 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.270455 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16904bf-0be6-4963-8ceb-c06342915cd3-kube-api-access-78kfb" (OuterVolumeSpecName: "kube-api-access-78kfb") pod "e16904bf-0be6-4963-8ceb-c06342915cd3" (UID: "e16904bf-0be6-4963-8ceb-c06342915cd3"). InnerVolumeSpecName "kube-api-access-78kfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:38.368531 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.368498 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78kfb\" (UniqueName: \"kubernetes.io/projected/e16904bf-0be6-4963-8ceb-c06342915cd3-kube-api-access-78kfb\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:38.368531 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.368524 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc47s\" (UniqueName: \"kubernetes.io/projected/772663fc-7928-467a-bc53-8d87e0ce6c68-kube-api-access-cc47s\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:38.368531 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:38.368534 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/772663fc-7928-467a-bc53-8d87e0ce6c68-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:39.160832 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.160800 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" Sep 10 16:00:39.161281 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.160834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w" event={"ID":"772663fc-7928-467a-bc53-8d87e0ce6c68","Type":"ContainerDied","Data":"a2a75b38f1fbbc4133119c8a215d4962b20fdfc5c4c2474c15788ce0342004fa"} Sep 10 16:00:39.161281 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.160893 2580 scope.go:117] "RemoveContainer" containerID="65da0d5fa4c83b45fadeb49ffd48924880abc740fb73c609b37eb813f74a4030" Sep 10 16:00:39.162286 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.162265 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" event={"ID":"e16904bf-0be6-4963-8ceb-c06342915cd3","Type":"ContainerDied","Data":"0b6c22fb5f9c26ce239cbbe8bac47f53f956a67c1285d42d130acc1c14997d81"} Sep 10 16:00:39.162286 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.162278 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd" Sep 10 16:00:39.169921 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.169904 2580 scope.go:117] "RemoveContainer" containerID="b909905ed3a4310b17d1f8c01ac896a1cabc7c432adfa3a2fdaf6e8f11aff98e" Sep 10 16:00:39.187388 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.187365 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w"] Sep 10 16:00:39.190540 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.190518 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f52559-6876c7579-5pg5w"] Sep 10 16:00:39.200407 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.200385 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd"] Sep 10 16:00:39.204558 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:39.204538 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f52559-894c95b46-pq7jd"] Sep 10 16:00:40.028144 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:40.028108 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772663fc-7928-467a-bc53-8d87e0ce6c68" path="/var/lib/kubelet/pods/772663fc-7928-467a-bc53-8d87e0ce6c68/volumes" Sep 10 16:00:40.028540 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:40.028527 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16904bf-0be6-4963-8ceb-c06342915cd3" path="/var/lib/kubelet/pods/e16904bf-0be6-4963-8ceb-c06342915cd3/volumes" Sep 10 16:00:44.700883 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.700798 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p"] Sep 10 16:00:44.701232 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701191 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772663fc-7928-467a-bc53-8d87e0ce6c68" containerName="payload-processing" Sep 10 16:00:44.701232 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701202 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="772663fc-7928-467a-bc53-8d87e0ce6c68" containerName="payload-processing" Sep 10 16:00:44.701232 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701213 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" Sep 10 16:00:44.701232 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701219 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" Sep 10 16:00:44.701232 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701230 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16904bf-0be6-4963-8ceb-c06342915cd3" containerName="payload-pre-processing" Sep 10 16:00:44.701232 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701235 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16904bf-0be6-4963-8ceb-c06342915cd3" containerName="payload-pre-processing" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701241 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e717b1-731a-4d65-8e02-32d95b23b00d" containerName="cleanup" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701246 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e717b1-731a-4d65-8e02-32d95b23b00d" containerName="cleanup" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701257 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31c9fd0-7c60-4391-a5dc-a7398af96780" containerName="cleanup" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701262 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31c9fd0-7c60-4391-a5dc-a7398af96780" containerName="cleanup" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701318 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701326 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="772663fc-7928-467a-bc53-8d87e0ce6c68" containerName="payload-processing" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701332 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31c9fd0-7c60-4391-a5dc-a7398af96780" containerName="cleanup" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701338 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701343 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e717b1-731a-4d65-8e02-32d95b23b00d" containerName="cleanup" Sep 10 16:00:44.701453 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.701350 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16904bf-0be6-4963-8ceb-c06342915cd3" containerName="payload-pre-processing" Sep 10 16:00:44.704661 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.704645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" Sep 10 16:00:44.707196 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.707177 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 16:00:44.707270 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.707247 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:00:44.711005 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.710971 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p"] Sep 10 16:00:44.827102 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.827067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f9822-b977-45be-936f-13b5f070952f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p\" (UID: \"3b7f9822-b977-45be-936f-13b5f070952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" Sep 10 16:00:44.927812 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.927768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f9822-b977-45be-936f-13b5f070952f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p\" (UID: \"3b7f9822-b977-45be-936f-13b5f070952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" Sep 10 16:00:44.928466 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:44.928447 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f9822-b977-45be-936f-13b5f070952f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p\" (UID: \"3b7f9822-b977-45be-936f-13b5f070952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" Sep 10 16:00:45.017604 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:45.017522 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" Sep 10 16:00:45.287951 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:45.287877 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p"] Sep 10 16:00:45.291468 ip-10-0-134-9 kubenswrapper[2580]: W0910 16:00:45.291434 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7f9822_b977_45be_936f_13b5f070952f.slice/crio-cb4b61f93d1f6f7a30af188249587f84ce92e2142f8539e043808e1ed5e802de WatchSource:0}: Error finding container cb4b61f93d1f6f7a30af188249587f84ce92e2142f8539e043808e1ed5e802de: Status 404 returned error can't find the container with id cb4b61f93d1f6f7a30af188249587f84ce92e2142f8539e043808e1ed5e802de Sep 10 16:00:46.204940 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:46.204839 2580 generic.go:356] "Generic (PLEG): container finished" podID="3b7f9822-b977-45be-936f-13b5f070952f" containerID="b543aa89e455126e00ab180f1f51a9ad5599801bca3bcbc7fbdaf8f23a62cc9f" exitCode=0 Sep 10 16:00:46.204940 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:46.204901 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" event={"ID":"3b7f9822-b977-45be-936f-13b5f070952f","Type":"ContainerDied","Data":"b543aa89e455126e00ab180f1f51a9ad5599801bca3bcbc7fbdaf8f23a62cc9f"} Sep 10 16:00:46.205374 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:46.204946 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" event={"ID":"3b7f9822-b977-45be-936f-13b5f070952f","Type":"ContainerStarted","Data":"cb4b61f93d1f6f7a30af188249587f84ce92e2142f8539e043808e1ed5e802de"} Sep 10 16:00:47.343970 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:47.343946 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" Sep 10 16:00:47.453078 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:47.453044 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f9822-b977-45be-936f-13b5f070952f-maas-api-service-ca\") pod \"3b7f9822-b977-45be-936f-13b5f070952f\" (UID: \"3b7f9822-b977-45be-936f-13b5f070952f\") " Sep 10 16:00:47.453472 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:47.453414 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7f9822-b977-45be-936f-13b5f070952f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b7f9822-b977-45be-936f-13b5f070952f" (UID: "3b7f9822-b977-45be-936f-13b5f070952f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:47.554769 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:47.554684 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b7f9822-b977-45be-936f-13b5f070952f-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:48.216165 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:48.216136 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" Sep 10 16:00:48.216328 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:48.216187 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p" event={"ID":"3b7f9822-b977-45be-936f-13b5f070952f","Type":"ContainerDied","Data":"cb4b61f93d1f6f7a30af188249587f84ce92e2142f8539e043808e1ed5e802de"} Sep 10 16:00:48.216328 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:48.216232 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb4b61f93d1f6f7a30af188249587f84ce92e2142f8539e043808e1ed5e802de" Sep 10 16:00:55.221461 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.221414 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn"] Sep 10 16:00:55.223742 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.223719 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6f8c71-29817600-6jgmn"] Sep 10 16:00:55.255233 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.255174 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j"] Sep 10 16:00:55.255472 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.255416 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" podUID="15e6016f-892e-466a-9e49-8afce7047256" containerName="payload-pre-processing" containerID="cri-o://6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee" gracePeriod=30 Sep 10 16:00:55.414693 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.414659 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p"] Sep 10 16:00:55.417611 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.417578 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6f8c71-wqx2p"] Sep 10 16:00:55.524873 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.524850 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 16:00:55.625884 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.625856 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e6016f-892e-466a-9e49-8afce7047256-plugins-config-volume\") pod \"15e6016f-892e-466a-9e49-8afce7047256\" (UID: \"15e6016f-892e-466a-9e49-8afce7047256\") " Sep 10 16:00:55.626068 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.625896 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnnp7\" (UniqueName: \"kubernetes.io/projected/15e6016f-892e-466a-9e49-8afce7047256-kube-api-access-cnnp7\") pod \"15e6016f-892e-466a-9e49-8afce7047256\" (UID: \"15e6016f-892e-466a-9e49-8afce7047256\") " Sep 10 16:00:55.626234 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.626203 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e6016f-892e-466a-9e49-8afce7047256-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15e6016f-892e-466a-9e49-8afce7047256" (UID: "15e6016f-892e-466a-9e49-8afce7047256"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:55.627911 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.627890 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e6016f-892e-466a-9e49-8afce7047256-kube-api-access-cnnp7" (OuterVolumeSpecName: "kube-api-access-cnnp7") pod "15e6016f-892e-466a-9e49-8afce7047256" (UID: "15e6016f-892e-466a-9e49-8afce7047256"). InnerVolumeSpecName "kube-api-access-cnnp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:55.727210 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.727177 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e6016f-892e-466a-9e49-8afce7047256-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:55.727210 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:55.727206 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnp7\" (UniqueName: \"kubernetes.io/projected/15e6016f-892e-466a-9e49-8afce7047256-kube-api-access-cnnp7\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:00:56.029360 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.029328 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31e717b1-731a-4d65-8e02-32d95b23b00d" path="/var/lib/kubelet/pods/31e717b1-731a-4d65-8e02-32d95b23b00d/volumes" Sep 10 16:00:56.029652 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.029638 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b7f9822-b977-45be-936f-13b5f070952f" path="/var/lib/kubelet/pods/3b7f9822-b977-45be-936f-13b5f070952f/volumes" Sep 10 16:00:56.261853 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.261817 2580 generic.go:356] "Generic (PLEG): container finished" podID="15e6016f-892e-466a-9e49-8afce7047256" containerID="6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee" exitCode=0 Sep 10 16:00:56.262288 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.261864 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" Sep 10 16:00:56.262288 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.261903 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" event={"ID":"15e6016f-892e-466a-9e49-8afce7047256","Type":"ContainerDied","Data":"6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee"} Sep 10 16:00:56.262288 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.261956 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j" event={"ID":"15e6016f-892e-466a-9e49-8afce7047256","Type":"ContainerDied","Data":"2eec615f4870dcef210eec4a15d79d377171ff7c8d9fd68ce5de3c76157c38a0"} Sep 10 16:00:56.262288 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.261977 2580 scope.go:117] "RemoveContainer" containerID="6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee" Sep 10 16:00:56.270964 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.270945 2580 scope.go:117] "RemoveContainer" containerID="6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee" Sep 10 16:00:56.271260 ip-10-0-134-9 kubenswrapper[2580]: E0910 16:00:56.271242 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee\": container with ID starting with 6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee not found: ID does not exist" containerID="6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee" Sep 10 16:00:56.271370 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.271267 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee"} err="failed to get container status \"6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee\": rpc error: code = NotFound desc = could not find container \"6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee\": container with ID starting with 6b57466dae8be789698497307f909e6eb84a430c84b0854fd15a15876a33fdee not found: ID does not exist" Sep 10 16:00:56.280664 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.280613 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j"] Sep 10 16:00:56.284289 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:56.284269 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6f8c71-98c94d54-mhn9j"] Sep 10 16:00:58.029352 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:00:58.029317 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15e6016f-892e-466a-9e49-8afce7047256" path="/var/lib/kubelet/pods/15e6016f-892e-466a-9e49-8afce7047256/volumes" Sep 10 16:01:02.304324 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.304289 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd"] Sep 10 16:01:02.304740 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.304689 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" Sep 10 16:01:02.304740 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.304701 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbdc0b48-61e0-4eda-b79b-b4c2a73c6d0b" containerName="istio-proxy" Sep 10 16:01:02.304740 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.304715 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b7f9822-b977-45be-936f-13b5f070952f" containerName="revoke-keys" Sep 10 16:01:02.304740 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.304721 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b7f9822-b977-45be-936f-13b5f070952f" containerName="revoke-keys" Sep 10 16:01:02.304740 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.304728 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e6016f-892e-466a-9e49-8afce7047256" containerName="payload-pre-processing" Sep 10 16:01:02.304740 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.304734 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e6016f-892e-466a-9e49-8afce7047256" containerName="payload-pre-processing" Sep 10 16:01:02.304995 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.304803 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e6016f-892e-466a-9e49-8afce7047256" containerName="payload-pre-processing" Sep 10 16:01:02.304995 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.304814 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b7f9822-b977-45be-936f-13b5f070952f" containerName="revoke-keys" Sep 10 16:01:02.309567 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.309549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" Sep 10 16:01:02.312265 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.312240 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 16:01:02.312733 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.312704 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vzj4n\"" Sep 10 16:01:02.316053 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.316031 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd"] Sep 10 16:01:02.489810 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.489772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4465dbb-3524-49d5-9d53-03ee62a8a354-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd\" (UID: \"d4465dbb-3524-49d5-9d53-03ee62a8a354\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" Sep 10 16:01:02.590659 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.590562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4465dbb-3524-49d5-9d53-03ee62a8a354-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd\" (UID: \"d4465dbb-3524-49d5-9d53-03ee62a8a354\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" Sep 10 16:01:02.591186 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.591163 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4465dbb-3524-49d5-9d53-03ee62a8a354-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd\" (UID: \"d4465dbb-3524-49d5-9d53-03ee62a8a354\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" Sep 10 16:01:02.623479 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.623447 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" Sep 10 16:01:02.896446 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:02.896402 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd"] Sep 10 16:01:02.899004 ip-10-0-134-9 kubenswrapper[2580]: W0910 16:01:02.898979 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4465dbb_3524_49d5_9d53_03ee62a8a354.slice/crio-a3444c77510933e5d926149716f9dae878e0bfa77bbd785531340defc9a4bd08 WatchSource:0}: Error finding container a3444c77510933e5d926149716f9dae878e0bfa77bbd785531340defc9a4bd08: Status 404 returned error can't find the container with id a3444c77510933e5d926149716f9dae878e0bfa77bbd785531340defc9a4bd08 Sep 10 16:01:03.302931 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:03.302883 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" event={"ID":"d4465dbb-3524-49d5-9d53-03ee62a8a354","Type":"ContainerStarted","Data":"d778ec62d356502ef042a7f12b7ae5c30872cb9f51cb3bf7c8ca49e84947afbe"} Sep 10 16:01:03.303110 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:03.302949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" event={"ID":"d4465dbb-3524-49d5-9d53-03ee62a8a354","Type":"ContainerStarted","Data":"a3444c77510933e5d926149716f9dae878e0bfa77bbd785531340defc9a4bd08"} Sep 10 16:01:03.317758 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:03.317707 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" podStartSLOduration=1.3176925320000001 podStartE2EDuration="1.317692532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:03.316509767 +0000 UTC m=+1721.878818967" watchObservedRunningTime="2026-09-10 16:01:03.317692532 +0000 UTC m=+1721.880001722" Sep 10 16:01:04.310805 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:04.310771 2580 generic.go:356] "Generic (PLEG): container finished" podID="d4465dbb-3524-49d5-9d53-03ee62a8a354" containerID="d778ec62d356502ef042a7f12b7ae5c30872cb9f51cb3bf7c8ca49e84947afbe" exitCode=0 Sep 10 16:01:04.310988 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:04.310837 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" event={"ID":"d4465dbb-3524-49d5-9d53-03ee62a8a354","Type":"ContainerDied","Data":"d778ec62d356502ef042a7f12b7ae5c30872cb9f51cb3bf7c8ca49e84947afbe"} Sep 10 16:01:05.445212 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:05.445190 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" Sep 10 16:01:05.621195 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:05.621167 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4465dbb-3524-49d5-9d53-03ee62a8a354-maas-api-service-ca\") pod \"d4465dbb-3524-49d5-9d53-03ee62a8a354\" (UID: \"d4465dbb-3524-49d5-9d53-03ee62a8a354\") " Sep 10 16:01:05.621525 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:05.621503 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4465dbb-3524-49d5-9d53-03ee62a8a354-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4465dbb-3524-49d5-9d53-03ee62a8a354" (UID: "d4465dbb-3524-49d5-9d53-03ee62a8a354"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:01:05.722343 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:05.722312 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4465dbb-3524-49d5-9d53-03ee62a8a354-maas-api-service-ca\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:01:06.322951 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:06.322866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" event={"ID":"d4465dbb-3524-49d5-9d53-03ee62a8a354","Type":"ContainerDied","Data":"a3444c77510933e5d926149716f9dae878e0bfa77bbd785531340defc9a4bd08"} Sep 10 16:01:06.322951 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:06.322890 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd" Sep 10 16:01:06.322951 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:06.322919 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3444c77510933e5d926149716f9dae878e0bfa77bbd785531340defc9a4bd08" Sep 10 16:01:12.900920 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:12.900883 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw"] Sep 10 16:01:12.901341 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:12.901106 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" podUID="bb38e674-04ab-4baf-8df9-f84f91af92f6" containerName="payload-processing" containerID="cri-o://d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c" gracePeriod=30 Sep 10 16:01:12.907451 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:12.907398 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk"] Sep 10 16:01:12.907819 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:12.907773 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" podUID="b76436ed-9465-40a1-8e35-c3664cea02f2" containerName="payload-pre-processing" containerID="cri-o://1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284" gracePeriod=30 Sep 10 16:01:13.065497 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.065448 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd"] Sep 10 16:01:13.067241 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.067211 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e412c4-wg2pd"] Sep 10 16:01:13.181471 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.181448 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 16:01:13.184311 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.184280 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 16:01:13.292637 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.292597 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb38e674-04ab-4baf-8df9-f84f91af92f6-plugins-config-volume\") pod \"bb38e674-04ab-4baf-8df9-f84f91af92f6\" (UID: \"bb38e674-04ab-4baf-8df9-f84f91af92f6\") " Sep 10 16:01:13.292830 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.292688 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzvtq\" (UniqueName: \"kubernetes.io/projected/b76436ed-9465-40a1-8e35-c3664cea02f2-kube-api-access-kzvtq\") pod \"b76436ed-9465-40a1-8e35-c3664cea02f2\" (UID: \"b76436ed-9465-40a1-8e35-c3664cea02f2\") " Sep 10 16:01:13.292830 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.292722 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhw5x\" (UniqueName: \"kubernetes.io/projected/bb38e674-04ab-4baf-8df9-f84f91af92f6-kube-api-access-jhw5x\") pod \"bb38e674-04ab-4baf-8df9-f84f91af92f6\" (UID: \"bb38e674-04ab-4baf-8df9-f84f91af92f6\") " Sep 10 16:01:13.292830 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.292767 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76436ed-9465-40a1-8e35-c3664cea02f2-plugins-config-volume\") pod \"b76436ed-9465-40a1-8e35-c3664cea02f2\" (UID: \"b76436ed-9465-40a1-8e35-c3664cea02f2\") " Sep 10 16:01:13.293032 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.293005 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb38e674-04ab-4baf-8df9-f84f91af92f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb38e674-04ab-4baf-8df9-f84f91af92f6" (UID: "bb38e674-04ab-4baf-8df9-f84f91af92f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:01:13.293214 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.293177 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76436ed-9465-40a1-8e35-c3664cea02f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b76436ed-9465-40a1-8e35-c3664cea02f2" (UID: "b76436ed-9465-40a1-8e35-c3664cea02f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:01:13.293329 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.293249 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb38e674-04ab-4baf-8df9-f84f91af92f6-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:01:13.294915 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.294892 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76436ed-9465-40a1-8e35-c3664cea02f2-kube-api-access-kzvtq" (OuterVolumeSpecName: "kube-api-access-kzvtq") pod "b76436ed-9465-40a1-8e35-c3664cea02f2" (UID: "b76436ed-9465-40a1-8e35-c3664cea02f2"). InnerVolumeSpecName "kube-api-access-kzvtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:01:13.294994 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.294941 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb38e674-04ab-4baf-8df9-f84f91af92f6-kube-api-access-jhw5x" (OuterVolumeSpecName: "kube-api-access-jhw5x") pod "bb38e674-04ab-4baf-8df9-f84f91af92f6" (UID: "bb38e674-04ab-4baf-8df9-f84f91af92f6"). InnerVolumeSpecName "kube-api-access-jhw5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:01:13.360018 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.359988 2580 generic.go:356] "Generic (PLEG): container finished" podID="b76436ed-9465-40a1-8e35-c3664cea02f2" containerID="1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284" exitCode=0 Sep 10 16:01:13.360178 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.360030 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" Sep 10 16:01:13.360178 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.360070 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" event={"ID":"b76436ed-9465-40a1-8e35-c3664cea02f2","Type":"ContainerDied","Data":"1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284"} Sep 10 16:01:13.360178 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.360124 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk" event={"ID":"b76436ed-9465-40a1-8e35-c3664cea02f2","Type":"ContainerDied","Data":"5163a2fe6e4c79115321263cc9d59179e38a1f62a262cbba3761666bafc4e9c3"} Sep 10 16:01:13.360178 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.360150 2580 scope.go:117] "RemoveContainer" containerID="1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284" Sep 10 16:01:13.361472 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.361453 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb38e674-04ab-4baf-8df9-f84f91af92f6" containerID="d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c" exitCode=0 Sep 10 16:01:13.361570 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.361493 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" Sep 10 16:01:13.361570 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.361545 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" event={"ID":"bb38e674-04ab-4baf-8df9-f84f91af92f6","Type":"ContainerDied","Data":"d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c"} Sep 10 16:01:13.361678 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.361610 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw" event={"ID":"bb38e674-04ab-4baf-8df9-f84f91af92f6","Type":"ContainerDied","Data":"7d727819071b34afd1a782edf8e14a27585df6f9f879f4acfdfeb97f0ee2f9a3"} Sep 10 16:01:13.371766 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.370950 2580 scope.go:117] "RemoveContainer" containerID="1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284" Sep 10 16:01:13.371766 ip-10-0-134-9 kubenswrapper[2580]: E0910 16:01:13.371281 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284\": container with ID starting with 1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284 not found: ID does not exist" containerID="1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284" Sep 10 16:01:13.371766 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.371315 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284"} err="failed to get container status \"1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284\": rpc error: code = NotFound desc = could not find container \"1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284\": container with ID starting with 1598dd4c1b53ae3ba1383739cd55d94f8da30cf6d2819e6bc3f474d847f8b284 not found: ID does not exist" Sep 10 16:01:13.371766 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.371338 2580 scope.go:117] "RemoveContainer" containerID="d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c" Sep 10 16:01:13.381775 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.381758 2580 scope.go:117] "RemoveContainer" containerID="d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c" Sep 10 16:01:13.382032 ip-10-0-134-9 kubenswrapper[2580]: E0910 16:01:13.382014 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c\": container with ID starting with d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c not found: ID does not exist" containerID="d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c" Sep 10 16:01:13.382079 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.382037 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c"} err="failed to get container status \"d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c\": rpc error: code = NotFound desc = could not find container \"d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c\": container with ID starting with d5f4b444b3f605ac789f13993095ef2450fcc11d6e3a1825bae6038de6c4683c not found: ID does not exist" Sep 10 16:01:13.389846 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.389824 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk"] Sep 10 16:01:13.393166 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.393144 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e412c4-69f68556cd-7jcgk"] Sep 10 16:01:13.393985 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.393964 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzvtq\" (UniqueName: \"kubernetes.io/projected/b76436ed-9465-40a1-8e35-c3664cea02f2-kube-api-access-kzvtq\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:01:13.394038 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.393992 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhw5x\" (UniqueName: \"kubernetes.io/projected/bb38e674-04ab-4baf-8df9-f84f91af92f6-kube-api-access-jhw5x\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:01:13.394038 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.394006 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b76436ed-9465-40a1-8e35-c3664cea02f2-plugins-config-volume\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:01:13.402994 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.402943 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw"] Sep 10 16:01:13.409353 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:13.409331 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e412c4-5d7d58b9dc-mg4sw"] Sep 10 16:01:14.028632 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:14.028602 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76436ed-9465-40a1-8e35-c3664cea02f2" path="/var/lib/kubelet/pods/b76436ed-9465-40a1-8e35-c3664cea02f2/volumes" Sep 10 16:01:14.028983 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:14.028946 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb38e674-04ab-4baf-8df9-f84f91af92f6" path="/var/lib/kubelet/pods/bb38e674-04ab-4baf-8df9-f84f91af92f6/volumes" Sep 10 16:01:14.029234 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:14.029222 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4465dbb-3524-49d5-9d53-03ee62a8a354" path="/var/lib/kubelet/pods/d4465dbb-3524-49d5-9d53-03ee62a8a354/volumes" Sep 10 16:01:22.180079 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.180047 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-569c478d95-cnfb2"] Sep 10 16:01:22.180439 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.180260 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-569c478d95-cnfb2" podUID="18070a4f-9f0a-4c1d-bd14-de83be91dc0a" containerName="manager" containerID="cri-o://b7d3c8ba3229c08f08605a87941099a953e3550fdff98d825e6f33d935105562" gracePeriod=30 Sep 10 16:01:22.412906 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.412872 2580 generic.go:356] "Generic (PLEG): container finished" podID="18070a4f-9f0a-4c1d-bd14-de83be91dc0a" containerID="b7d3c8ba3229c08f08605a87941099a953e3550fdff98d825e6f33d935105562" exitCode=0 Sep 10 16:01:22.412987 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.412935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-cnfb2" event={"ID":"18070a4f-9f0a-4c1d-bd14-de83be91dc0a","Type":"ContainerDied","Data":"b7d3c8ba3229c08f08605a87941099a953e3550fdff98d825e6f33d935105562"} Sep 10 16:01:22.434134 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.434079 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 16:01:22.481855 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.481824 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-metrics-certs\") pod \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " Sep 10 16:01:22.482034 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.481957 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-webhook-cert\") pod \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " Sep 10 16:01:22.482034 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.482009 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg5f9\" (UniqueName: \"kubernetes.io/projected/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-kube-api-access-cg5f9\") pod \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\" (UID: \"18070a4f-9f0a-4c1d-bd14-de83be91dc0a\") " Sep 10 16:01:22.484108 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.484078 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "18070a4f-9f0a-4c1d-bd14-de83be91dc0a" (UID: "18070a4f-9f0a-4c1d-bd14-de83be91dc0a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:01:22.484410 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.484382 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "18070a4f-9f0a-4c1d-bd14-de83be91dc0a" (UID: "18070a4f-9f0a-4c1d-bd14-de83be91dc0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:01:22.484548 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.484412 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-kube-api-access-cg5f9" (OuterVolumeSpecName: "kube-api-access-cg5f9") pod "18070a4f-9f0a-4c1d-bd14-de83be91dc0a" (UID: "18070a4f-9f0a-4c1d-bd14-de83be91dc0a"). InnerVolumeSpecName "kube-api-access-cg5f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:01:22.583049 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.583014 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-webhook-cert\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:01:22.583049 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.583042 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg5f9\" (UniqueName: \"kubernetes.io/projected/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-kube-api-access-cg5f9\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:01:22.583049 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:22.583052 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18070a4f-9f0a-4c1d-bd14-de83be91dc0a-metrics-certs\") on node \"ip-10-0-134-9.ec2.internal\" DevicePath \"\"" Sep 10 16:01:23.419458 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:23.419416 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-cnfb2" Sep 10 16:01:23.419458 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:23.419453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-cnfb2" event={"ID":"18070a4f-9f0a-4c1d-bd14-de83be91dc0a","Type":"ContainerDied","Data":"aca24721cdeef506fe7670ab361a9169aad63457757304aeb7a34f93a5698f4a"} Sep 10 16:01:23.419949 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:23.419512 2580 scope.go:117] "RemoveContainer" containerID="b7d3c8ba3229c08f08605a87941099a953e3550fdff98d825e6f33d935105562" Sep 10 16:01:23.439983 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:23.439958 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-569c478d95-cnfb2"] Sep 10 16:01:23.443724 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:23.443701 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-569c478d95-cnfb2"] Sep 10 16:01:24.028994 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:01:24.028963 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18070a4f-9f0a-4c1d-bd14-de83be91dc0a" path="/var/lib/kubelet/pods/18070a4f-9f0a-4c1d-bd14-de83be91dc0a/volumes" Sep 10 16:02:22.105078 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:02:22.105049 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 16:02:22.108381 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:02:22.108358 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 16:02:22.433405 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:02:22.433377 2580 scope.go:117] "RemoveContainer" containerID="bf9347470feacbce5cfda92f08ddc242395e1af7b8f5571a85470330e9cb7be7" Sep 10 16:02:22.441591 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:02:22.441571 2580 scope.go:117] "RemoveContainer" containerID="1cc42f5f4ebe752578a582c0f43624748e7bf467a3c817c3d1e130d578f36fbf" Sep 10 16:03:22.470466 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:03:22.470431 2580 scope.go:117] "RemoveContainer" containerID="d3431925807dd55e22e4718db92d8b9b271dd33fb3c7bb4c0ed0959caa60ca30" Sep 10 16:03:22.478593 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:03:22.478570 2580 scope.go:117] "RemoveContainer" containerID="11bc5ed8105a5b6c77b151e42d2ccca2c6e7211850758649baa8bd1aa6095e9e" Sep 10 16:05:22.507762 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:05:22.507686 2580 scope.go:117] "RemoveContainer" containerID="d8e92de448c4fa58fab49bfcf9aa47e83dcd93ad9884d81b2545a302c1ffbe6b" Sep 10 16:06:22.528499 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:06:22.528467 2580 scope.go:117] "RemoveContainer" containerID="d4563042551f803783a1c04e0cbb527b1f716d1c7bbaa8d3a37d8127f7363379" Sep 10 16:06:22.536572 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:06:22.536554 2580 scope.go:117] "RemoveContainer" containerID="c637414233e23fa0f81a83a391cc75bdcfba9bc3f97ddc167090a6db25930f68" Sep 10 16:06:22.543710 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:06:22.543687 2580 scope.go:117] "RemoveContainer" containerID="eb645a72da420d39506ad721c75581a0f4c470b36e0f6360388a28ab4c540d43" Sep 10 16:07:22.134864 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:22.134830 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 16:07:22.145906 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:22.145884 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 16:07:22.568588 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:22.568504 2580 scope.go:117] "RemoveContainer" containerID="d778ec62d356502ef042a7f12b7ae5c30872cb9f51cb3bf7c8ca49e84947afbe" Sep 10 16:07:22.576598 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:22.576577 2580 scope.go:117] "RemoveContainer" containerID="b543aa89e455126e00ab180f1f51a9ad5599801bca3bcbc7fbdaf8f23a62cc9f" Sep 10 16:07:37.367965 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:37.367936 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2dncm_0e3d9454-6af8-4bea-9207-96ebd23b3bce/server/0.log" Sep 10 16:07:37.483390 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:37.483360 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sgxwt_63fd10e0-82f9-444c-8302-2a31c8c1b98f/manager/0.log" Sep 10 16:07:37.711144 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:37.711057 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f4cd445f-bv7vh_0180c692-a6f5-460f-a229-a6679ec710a4/copy-manifests/0.log" Sep 10 16:07:37.722227 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:37.722203 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f4cd445f-bv7vh_0180c692-a6f5-460f-a229-a6679ec710a4/manager/0.log" Sep 10 16:07:38.445286 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:38.445255 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817585-fcvrd_ec595964-a1b9-4d9b-accd-23b5cedf3773/cleanup/0.log" Sep 10 16:07:40.592691 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:40.592658 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c54rwr_7c0b58fd-8f53-4a8a-ab2e-fd948778287d/istio-proxy/0.log" Sep 10 16:07:40.720939 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:40.720910 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zbzjr_7066d53c-5673-443c-bd2f-2dbdc3059bdd/discovery/0.log" Sep 10 16:07:41.065046 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:41.065015 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k7z58_60f42cf8-4bed-4d27-8d4e-766522ecb76c/istio-proxy/1.log" Sep 10 16:07:41.319490 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:41.319395 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qb9kv_fe46e4b2-e73b-4192-a133-bc065eb7f948/payload-processing/0.log" Sep 10 16:07:41.917622 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:41.917595 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-lsrhc_6a95b701-9838-4162-a928-0d96dbb6f938/main/0.log" Sep 10 16:07:42.159778 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:42.159748 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-qrv26_994554d5-16b5-4cf4-a8ad-e0e94f16da9b/main/0.log" Sep 10 16:07:42.394097 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:42.394071 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-c9tcs_9c5060a8-b260-4aea-8710-7c53f612b8b0/main/0.log" Sep 10 16:07:51.351017 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:51.350989 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8ld8d_f044f5a4-1d02-48b3-a5c3-0dc9f2212f51/global-pull-secret-syncer/0.log" Sep 10 16:07:51.508797 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:51.508769 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rzbl8_fed4497c-aa2c-4f87-a183-ac8d193d9e93/konnectivity-agent/0.log" Sep 10 16:07:51.529761 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:51.529725 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-9.ec2.internal_3cdfc09abde163e5efcd6f1fa3f633fc/haproxy/0.log" Sep 10 16:07:58.375317 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.375285 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8hgk_59d30f05-1cd1-4e1e-9f2c-31d142c62bfc/node-exporter/0.log" Sep 10 16:07:58.396925 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.396902 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8hgk_59d30f05-1cd1-4e1e-9f2c-31d142c62bfc/kube-rbac-proxy/0.log" Sep 10 16:07:58.422581 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.422554 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8hgk_59d30f05-1cd1-4e1e-9f2c-31d142c62bfc/init-textfile/0.log" Sep 10 16:07:58.674035 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.674010 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b126c126-5a9c-4087-8432-25962283b92b/prometheus/0.log" Sep 10 16:07:58.695941 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.695915 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b126c126-5a9c-4087-8432-25962283b92b/config-reloader/0.log" Sep 10 16:07:58.721167 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.721145 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b126c126-5a9c-4087-8432-25962283b92b/thanos-sidecar/0.log" Sep 10 16:07:58.746240 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.746217 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b126c126-5a9c-4087-8432-25962283b92b/kube-rbac-proxy-web/0.log" Sep 10 16:07:58.771096 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.771068 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b126c126-5a9c-4087-8432-25962283b92b/kube-rbac-proxy/0.log" Sep 10 16:07:58.791190 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.791162 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b126c126-5a9c-4087-8432-25962283b92b/kube-rbac-proxy-thanos/0.log" Sep 10 16:07:58.813060 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.813035 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b126c126-5a9c-4087-8432-25962283b92b/init-config-reloader/0.log" Sep 10 16:07:58.840455 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.840414 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-slm9x_763bec0e-dd29-42ee-b84a-83684e5b358c/prometheus-operator/0.log" Sep 10 16:07:58.859138 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.859111 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-slm9x_763bec0e-dd29-42ee-b84a-83684e5b358c/kube-rbac-proxy/0.log" Sep 10 16:07:58.928788 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.928712 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d6d6544c7-rdh7d_8f0ca232-2cd0-4ea1-ab3d-426d08c833b9/telemeter-client/0.log" Sep 10 16:07:58.953614 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.953589 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d6d6544c7-rdh7d_8f0ca232-2cd0-4ea1-ab3d-426d08c833b9/reload/0.log" Sep 10 16:07:58.975594 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:07:58.975564 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d6d6544c7-rdh7d_8f0ca232-2cd0-4ea1-ab3d-426d08c833b9/kube-rbac-proxy/0.log" Sep 10 16:08:00.505298 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:00.505268 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-5m5tq_909ac02b-6554-404c-8b05-572d1d5902a9/networking-console-plugin/0.log" Sep 10 16:08:01.083103 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.083072 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/1.log" Sep 10 16:08:01.087587 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.087567 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-zkn28_ecc661e5-071a-44c4-a08d-7cf930601269/console-operator/2.log" Sep 10 16:08:01.337548 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337474 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488"] Sep 10 16:08:01.337868 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337849 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76436ed-9465-40a1-8e35-c3664cea02f2" containerName="payload-pre-processing" Sep 10 16:08:01.337868 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337865 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76436ed-9465-40a1-8e35-c3664cea02f2" containerName="payload-pre-processing" Sep 10 16:08:01.337868 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337875 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb38e674-04ab-4baf-8df9-f84f91af92f6" containerName="payload-processing" Sep 10 16:08:01.338107 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337881 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb38e674-04ab-4baf-8df9-f84f91af92f6" containerName="payload-processing" Sep 10 16:08:01.338107 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337899 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18070a4f-9f0a-4c1d-bd14-de83be91dc0a" containerName="manager" Sep 10 16:08:01.338107 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337908 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="18070a4f-9f0a-4c1d-bd14-de83be91dc0a" containerName="manager" Sep 10 16:08:01.338107 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337920 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4465dbb-3524-49d5-9d53-03ee62a8a354" containerName="revoke-keys" Sep 10 16:08:01.338107 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337928 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4465dbb-3524-49d5-9d53-03ee62a8a354" containerName="revoke-keys" Sep 10 16:08:01.338107 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.337990 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb38e674-04ab-4baf-8df9-f84f91af92f6" containerName="payload-processing" Sep 10 16:08:01.338107 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.338007 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="18070a4f-9f0a-4c1d-bd14-de83be91dc0a" containerName="manager" Sep 10 16:08:01.338107 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.338016 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4465dbb-3524-49d5-9d53-03ee62a8a354" containerName="revoke-keys" Sep 10 16:08:01.338107 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.338023 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76436ed-9465-40a1-8e35-c3664cea02f2" containerName="payload-pre-processing" Sep 10 16:08:01.341227 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.341210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.343474 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.343458 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ccz5g\"/\"openshift-service-ca.crt\"" Sep 10 16:08:01.344539 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.344518 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ccz5g\"/\"kube-root-ca.crt\"" Sep 10 16:08:01.344650 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.344549 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ccz5g\"/\"default-dockercfg-5k4tw\"" Sep 10 16:08:01.349273 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.349243 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488"] Sep 10 16:08:01.456856 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.456809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghdl\" (UniqueName: \"kubernetes.io/projected/2ce9e7c9-6980-43a1-9561-f502b1f2455b-kube-api-access-cghdl\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.456856 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.456859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-podres\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.457077 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.456987 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-sys\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.457077 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.457037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-proc\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.457151 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.457081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-lib-modules\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.557837 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.557794 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghdl\" (UniqueName: \"kubernetes.io/projected/2ce9e7c9-6980-43a1-9561-f502b1f2455b-kube-api-access-cghdl\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.558276 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.557845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-podres\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.558276 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.557910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-sys\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.558276 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.557948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-proc\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.558276 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.557971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-lib-modules\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.558276 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.558056 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-proc\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.558276 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.558057 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-sys\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.558276 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.558054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-podres\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.558276 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.558139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ce9e7c9-6980-43a1-9561-f502b1f2455b-lib-modules\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.565151 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.565128 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghdl\" (UniqueName: \"kubernetes.io/projected/2ce9e7c9-6980-43a1-9561-f502b1f2455b-kube-api-access-cghdl\") pod \"perf-node-gather-daemonset-cs488\" (UID: \"2ce9e7c9-6980-43a1-9561-f502b1f2455b\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.651942 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.651905 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:01.708658 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.708585 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-bzktv_a73adce0-f5ba-43c0-973d-ced40480a9a9/download-server/0.log" Sep 10 16:08:01.924187 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:01.924112 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488"] Sep 10 16:08:01.926870 ip-10-0-134-9 kubenswrapper[2580]: W0910 16:08:01.926824 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2ce9e7c9_6980_43a1_9561_f502b1f2455b.slice/crio-344f3e7077ae9e51349b6c5b82b9e79c2f28c9db76e20fc3d8095c53c3681677 WatchSource:0}: Error finding container 344f3e7077ae9e51349b6c5b82b9e79c2f28c9db76e20fc3d8095c53c3681677: Status 404 returned error can't find the container with id 344f3e7077ae9e51349b6c5b82b9e79c2f28c9db76e20fc3d8095c53c3681677 Sep 10 16:08:02.401001 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:02.400974 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-cvzzz_ab259f57-795d-463b-b058-ccfe8d326b3d/volume-data-source-validator/0.log" Sep 10 16:08:02.416206 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:02.416175 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" event={"ID":"2ce9e7c9-6980-43a1-9561-f502b1f2455b","Type":"ContainerStarted","Data":"4c717e7730c6d13085b8d8a9737509affbec380bb84052394d85f3fc8461f58f"} Sep 10 16:08:02.416357 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:02.416234 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" event={"ID":"2ce9e7c9-6980-43a1-9561-f502b1f2455b","Type":"ContainerStarted","Data":"344f3e7077ae9e51349b6c5b82b9e79c2f28c9db76e20fc3d8095c53c3681677"} Sep 10 16:08:02.416357 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:02.416265 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:02.432237 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:02.432186 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" podStartSLOduration=1.432171055 podStartE2EDuration="1.432171055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:02.430118902 +0000 UTC m=+2140.992428091" watchObservedRunningTime="2026-09-10 16:08:02.432171055 +0000 UTC m=+2140.994480274" Sep 10 16:08:03.644756 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:03.644730 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m5dct_fa218d1e-4464-49e6-b1e8-141f0dd3aace/dns/0.log" Sep 10 16:08:03.665354 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:03.665331 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m5dct_fa218d1e-4464-49e6-b1e8-141f0dd3aace/kube-rbac-proxy/0.log" Sep 10 16:08:03.751753 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:03.751728 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hvffx_f636588a-67d1-4196-91d5-783f5ca11b8a/dns-node-resolver/0.log" Sep 10 16:08:04.266505 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:04.266475 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-596fd9b478-qwskh_b04946a6-b69a-4e35-8790-c656d7747513/registry/0.log" Sep 10 16:08:04.285865 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:04.285843 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6hvnc_b92f14a2-c880-4843-98be-7a8480914160/node-ca/0.log" Sep 10 16:08:05.369225 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:05.369196 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c54rwr_7c0b58fd-8f53-4a8a-ab2e-fd948778287d/istio-proxy/0.log" Sep 10 16:08:05.598508 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:05.598460 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zbzjr_7066d53c-5673-443c-bd2f-2dbdc3059bdd/discovery/0.log" Sep 10 16:08:05.678695 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:05.678621 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k7z58_60f42cf8-4bed-4d27-8d4e-766522ecb76c/istio-proxy/0.log" Sep 10 16:08:05.686797 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:05.686772 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k7z58_60f42cf8-4bed-4d27-8d4e-766522ecb76c/istio-proxy/1.log" Sep 10 16:08:05.768946 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:05.768922 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qb9kv_fe46e4b2-e73b-4192-a133-bc065eb7f948/payload-processing/0.log" Sep 10 16:08:06.730617 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:06.730570 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-htnn6_9c0fc599-85ef-4ad2-a3d6-ed614c8c6fbd/serve-healthcheck-canary/0.log" Sep 10 16:08:07.390988 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:07.390960 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w8np8_028c54f4-e5fc-4f58-9127-390792d644ef/kube-rbac-proxy/0.log" Sep 10 16:08:07.412268 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:07.412247 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w8np8_028c54f4-e5fc-4f58-9127-390792d644ef/exporter/0.log" Sep 10 16:08:07.435162 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:07.435137 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w8np8_028c54f4-e5fc-4f58-9127-390792d644ef/extractor/0.log" Sep 10 16:08:08.431465 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:08.431417 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-cs488" Sep 10 16:08:09.927516 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:09.927486 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-2dncm_0e3d9454-6af8-4bea-9207-96ebd23b3bce/server/0.log" Sep 10 16:08:10.004568 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:10.004539 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sgxwt_63fd10e0-82f9-444c-8302-2a31c8c1b98f/manager/0.log" Sep 10 16:08:10.128467 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:10.128414 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f4cd445f-bv7vh_0180c692-a6f5-460f-a229-a6679ec710a4/manager/0.log" Sep 10 16:08:10.147674 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:10.147647 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f4cd445f-bv7vh_0180c692-a6f5-460f-a229-a6679ec710a4/copy-manifests/0.log" Sep 10 16:08:11.467593 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:11.467562 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-556d846896-p5gdn_a9b2ec53-fc10-49e7-8ba7-3260500777a3/manager/0.log" Sep 10 16:08:12.114712 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:12.114678 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c54rwr_7c0b58fd-8f53-4a8a-ab2e-fd948778287d/istio-proxy/0.log" Sep 10 16:08:12.330325 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:12.330247 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zbzjr_7066d53c-5673-443c-bd2f-2dbdc3059bdd/discovery/0.log" Sep 10 16:08:12.382211 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:12.382181 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k7z58_60f42cf8-4bed-4d27-8d4e-766522ecb76c/istio-proxy/0.log" Sep 10 16:08:12.390588 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:12.390563 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k7z58_60f42cf8-4bed-4d27-8d4e-766522ecb76c/istio-proxy/1.log" Sep 10 16:08:12.453957 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:12.453931 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qb9kv_fe46e4b2-e73b-4192-a133-bc065eb7f948/payload-processing/0.log" Sep 10 16:08:14.035271 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:14.035189 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799c54rwr_7c0b58fd-8f53-4a8a-ab2e-fd948778287d/istio-proxy/0.log" Sep 10 16:08:14.252041 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:14.252011 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zbzjr_7066d53c-5673-443c-bd2f-2dbdc3059bdd/discovery/0.log" Sep 10 16:08:14.304563 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:14.304475 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k7z58_60f42cf8-4bed-4d27-8d4e-766522ecb76c/istio-proxy/0.log" Sep 10 16:08:14.312067 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:14.312045 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-k7z58_60f42cf8-4bed-4d27-8d4e-766522ecb76c/istio-proxy/1.log" Sep 10 16:08:14.376040 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:14.376013 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qb9kv_fe46e4b2-e73b-4192-a133-bc065eb7f948/payload-processing/0.log" Sep 10 16:08:16.803615 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:16.803591 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-wv5ph_325fcde8-67ae-4bfa-97bd-f6166cc1471d/migrator/0.log" Sep 10 16:08:16.823136 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:16.823110 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-wv5ph_325fcde8-67ae-4bfa-97bd-f6166cc1471d/graceful-termination/0.log" Sep 10 16:08:18.567559 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.567476 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xb459_368dd85e-db5c-42da-b98c-b8cd54960d1b/kube-multus-additional-cni-plugins/0.log" Sep 10 16:08:18.587263 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.587232 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xb459_368dd85e-db5c-42da-b98c-b8cd54960d1b/egress-router-binary-copy/0.log" Sep 10 16:08:18.606886 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.606859 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xb459_368dd85e-db5c-42da-b98c-b8cd54960d1b/cni-plugins/0.log" Sep 10 16:08:18.625551 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.625515 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xb459_368dd85e-db5c-42da-b98c-b8cd54960d1b/bond-cni-plugin/0.log" Sep 10 16:08:18.643965 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.643937 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xb459_368dd85e-db5c-42da-b98c-b8cd54960d1b/routeoverride-cni/0.log" Sep 10 16:08:18.664250 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.664230 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xb459_368dd85e-db5c-42da-b98c-b8cd54960d1b/whereabouts-cni-bincopy/0.log" Sep 10 16:08:18.683334 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.683313 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xb459_368dd85e-db5c-42da-b98c-b8cd54960d1b/whereabouts-cni/0.log" Sep 10 16:08:18.726068 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.726046 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jh65b_baa7bbbd-ba07-41eb-883f-4fa765f40091/kube-multus/0.log" Sep 10 16:08:18.899316 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.899288 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hvz5r_6150e082-626a-47a0-b8a0-5a9ea64b8251/network-metrics-daemon/0.log" Sep 10 16:08:18.918143 ip-10-0-134-9 kubenswrapper[2580]: I0910 16:08:18.918120 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hvz5r_6150e082-626a-47a0-b8a0-5a9ea64b8251/kube-rbac-proxy/0.log"