Sep 10 17:11:16.425318 ip-10-0-133-191 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 17:11:16.425332 ip-10-0-133-191 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 17:11:16.425341 ip-10-0-133-191 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 17:11:16.425546 ip-10-0-133-191 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 17:11:27.797294 ip-10-0-133-191 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 17:11:27.797311 ip-10-0-133-191 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 811e3b750d0b4327bacca15934073d29 -- Sep 10 17:13:20.784345 ip-10-0-133-191 systemd[1]: Starting Kubernetes Kubelet... Sep 10 17:13:21.180380 ip-10-0-133-191 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 17:13:21.180380 ip-10-0-133-191 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 17:13:21.180380 ip-10-0-133-191 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 17:13:21.180380 ip-10-0-133-191 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 17:13:21.180380 ip-10-0-133-191 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 17:13:21.181338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.181251 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 17:13:21.185232 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185215 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:13:21.185232 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185232 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185236 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185239 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185244 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185246 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185251 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185255 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185259 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185262 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185264 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185267 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185270 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185272 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185275 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185277 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185280 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185287 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185289 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185292 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185295 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:13:21.185297 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185297 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185301 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185303 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185306 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185309 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185312 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185315 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185317 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185320 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185323 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185325 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185328 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185330 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185333 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185337 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185340 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185343 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185346 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185348 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:13:21.185796 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185351 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185353 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185356 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185358 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185360 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185363 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185365 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185369 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185372 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185374 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185376 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185379 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185381 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185384 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185386 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185389 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185392 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185394 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185397 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185399 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:13:21.186282 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185402 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185404 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185407 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185409 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185412 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185414 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185417 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185419 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185422 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185424 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185427 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185429 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185431 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185434 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185436 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185446 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185448 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185451 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185454 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:13:21.186896 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185457 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185459 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185462 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185465 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185467 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185470 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185472 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185475 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185479 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185482 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185489 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185491 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185494 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185496 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185499 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185503 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185507 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185510 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185513 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:13:21.187381 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.185516 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186212 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186219 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186224 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186226 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186229 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186232 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186234 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186237 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186240 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186242 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186245 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186248 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186250 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186253 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186257 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186262 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186265 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186267 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:13:21.187851 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186270 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186273 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186277 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186280 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186283 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186286 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186289 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186291 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186294 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186297 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186300 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186302 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186305 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186307 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186310 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186312 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186315 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186318 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186320 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186322 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:13:21.188355 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186325 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186327 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186330 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186332 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186335 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186338 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186341 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186344 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186347 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186349 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186352 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186354 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186357 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186359 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186362 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186365 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186367 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186370 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186372 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186375 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:13:21.188832 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186377 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186380 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186382 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186384 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186387 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186390 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186392 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186395 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186398 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186400 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186402 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186405 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186407 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186412 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186414 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186416 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186419 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186421 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186424 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186426 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:13:21.189335 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186430 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186434 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186438 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186441 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186444 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186447 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186450 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186452 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186455 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186457 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186459 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186462 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186465 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186467 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186471 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186475 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186478 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186481 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186483 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:13:21.189837 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186486 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186488 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186852 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186856 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186860 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186862 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186865 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186868 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186870 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186873 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186876 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186878 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186882 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186886 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186890 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186893 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186896 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186902 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186905 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186908 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:13:21.190331 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186911 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186913 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186917 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186921 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186924 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186926 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186929 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186932 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186935 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186937 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186940 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186942 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186945 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186948 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186950 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186953 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186955 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186958 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186960 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:13:21.190829 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186963 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186966 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186968 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186971 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186974 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186976 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186980 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186982 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186985 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186988 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186995 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.186997 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187000 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187002 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187005 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187008 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187010 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187013 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187016 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187018 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:13:21.191309 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187021 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187023 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187026 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187028 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187047 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187049 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187052 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187055 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187058 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187061 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187064 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187066 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187069 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187071 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187074 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187076 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187079 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187082 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187085 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187088 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:13:21.191841 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187091 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187094 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187098 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187101 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187104 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187106 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187109 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187112 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187116 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187119 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187122 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187125 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187127 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187130 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187133 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187136 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187138 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187141 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187143 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:13:21.192354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187146 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187148 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187151 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187224 2557 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187273 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187281 2557 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187286 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187290 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187293 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187297 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187302 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187305 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187308 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187312 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187315 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187321 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187324 2557 flags.go:64] FLAG: --cgroup-root="" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187326 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187329 2557 flags.go:64] FLAG: --client-ca-file="" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187332 2557 flags.go:64] FLAG: --cloud-provider="external" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187335 2557 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187339 2557 flags.go:64] FLAG: --cluster-domain="" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187342 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187345 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 17:13:21.192821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187348 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187351 2557 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187355 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187358 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187361 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187364 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187367 2557 flags.go:64] FLAG: --contention-profiling="false" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187370 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187373 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187376 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187379 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187383 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187386 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187389 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187392 2557 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187395 2557 flags.go:64] FLAG: --enable-server="true" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187398 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187403 2557 flags.go:64] FLAG: --event-burst="100" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187406 2557 flags.go:64] FLAG: --event-qps="50" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187409 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187412 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187415 2557 flags.go:64] FLAG: --eviction-hard="" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187419 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187423 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187426 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 17:13:21.193431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187429 2557 flags.go:64] FLAG: --eviction-soft="" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187432 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187435 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187438 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187441 2557 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187444 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187447 2557 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187449 2557 flags.go:64] FLAG: --feature-gates="" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187453 2557 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187456 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187459 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187462 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187467 2557 flags.go:64] FLAG: --healthz-port="10248" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187471 2557 flags.go:64] FLAG: --help="false" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187473 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187477 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187480 2557 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187483 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187487 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187490 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187493 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187496 2557 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187499 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 17:13:21.194100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187502 2557 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187505 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187508 2557 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187510 2557 flags.go:64] FLAG: --kube-reserved="" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187513 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187516 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187519 2557 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187523 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187526 2557 flags.go:64] FLAG: --lock-file="" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187529 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187533 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187536 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187541 2557 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187544 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187547 2557 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187549 2557 flags.go:64] FLAG: --logging-format="text" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187552 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187556 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187559 2557 flags.go:64] FLAG: --manifest-url="" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187561 2557 flags.go:64] FLAG: --manifest-url-header="" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187565 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187570 2557 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187574 2557 flags.go:64] FLAG: --max-pods="110" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187577 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187580 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 17:13:21.194705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187583 2557 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187586 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187589 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187592 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187595 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187602 2557 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187605 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187608 2557 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187611 2557 flags.go:64] FLAG: --pod-cidr="" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187614 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187619 2557 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187622 2557 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187625 2557 flags.go:64] FLAG: --pods-per-core="0" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187628 2557 flags.go:64] FLAG: --port="10250" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187631 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187635 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d20b17a389436abf" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187638 2557 flags.go:64] FLAG: --qos-reserved="" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187641 2557 flags.go:64] FLAG: --read-only-port="10255" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187644 2557 flags.go:64] FLAG: --register-node="true" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187647 2557 flags.go:64] FLAG: --register-with-taints="" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187650 2557 flags.go:64] FLAG: --registry-burst="10" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187653 2557 flags.go:64] FLAG: --registry-qps="5" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187656 2557 flags.go:64] FLAG: --reserved-cpus="" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187659 2557 flags.go:64] FLAG: --reserved-memory="" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187662 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 17:13:21.195371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187665 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187668 2557 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187671 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187674 2557 flags.go:64] FLAG: --runonce="false" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187679 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187682 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187684 2557 flags.go:64] FLAG: --seccomp-default="false" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187687 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187690 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187693 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187696 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187699 2557 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187702 2557 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187705 2557 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187708 2557 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187711 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187714 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187717 2557 flags.go:64] FLAG: --system-cgroups="" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187720 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187725 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187728 2557 flags.go:64] FLAG: --tls-cert-file="" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187731 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187734 2557 flags.go:64] FLAG: --tls-min-version="" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187738 2557 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187741 2557 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 17:13:21.195991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187743 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187746 2557 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187749 2557 flags.go:64] FLAG: --v="2" Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187757 2557 flags.go:64] FLAG: --version="false" Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187761 2557 flags.go:64] FLAG: --vmodule="" Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187765 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.187768 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187894 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187898 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187901 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187904 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187908 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187910 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187913 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187915 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187918 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187920 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187923 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187925 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187929 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187931 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:13:21.196632 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187934 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187936 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187939 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187941 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187944 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187948 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187951 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187955 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187958 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187961 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187963 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187966 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187969 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187971 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187974 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187977 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187979 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187981 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187985 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:13:21.197180 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187987 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187990 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187992 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187996 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.187999 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188003 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188007 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188011 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188015 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188017 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188021 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188023 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188026 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188028 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188044 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188047 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188050 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188053 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188056 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188058 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:13:21.197657 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188062 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188065 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188068 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188070 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188073 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188076 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188080 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188083 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188086 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188089 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188091 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188095 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188097 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188100 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188102 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188106 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188109 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188111 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188114 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188116 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:13:21.198235 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188119 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188122 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188126 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188129 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188133 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188136 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188138 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188141 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188144 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188147 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188150 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188152 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188158 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188161 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188164 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188166 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188169 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188172 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188174 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188177 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:13:21.198723 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188180 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188182 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188185 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188187 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188190 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.188192 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.188197 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.195668 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.195805 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.195819 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.195825 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195888 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195894 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195899 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:13:21.199239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195903 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195907 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195910 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195912 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195915 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195918 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195920 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195923 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195927 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195930 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195934 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195937 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195940 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195942 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195945 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195948 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195951 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195954 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195956 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:13:21.199605 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195959 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195961 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195964 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195967 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195969 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195972 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195975 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195983 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195987 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195990 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195992 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195995 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.195998 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196001 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196003 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196006 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196009 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196011 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196014 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:13:21.200087 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196016 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196019 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196021 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196024 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196026 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196029 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196047 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196051 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196053 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196056 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196059 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196061 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196064 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196066 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196069 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196071 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196075 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196079 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196082 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196084 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:13:21.200564 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196088 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196097 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196100 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196103 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196106 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196108 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196111 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196113 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196116 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196119 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196121 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196124 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196126 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196129 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196132 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196134 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196136 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196139 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196142 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196144 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:13:21.201076 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196147 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196149 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196152 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196154 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196157 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196159 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196161 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196164 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196166 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196169 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196172 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196174 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196177 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196180 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196188 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196191 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196193 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:13:21.201572 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196195 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.196201 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196386 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196394 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196397 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196400 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196403 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196406 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196408 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196411 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196413 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196416 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196418 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196421 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196423 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 17:13:21.202101 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196426 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196428 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196430 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196433 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196436 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196438 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196440 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196443 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196445 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196448 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196451 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196453 2557 feature_gate.go:328] unrecognized feature gate: Example Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196456 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196459 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196462 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196474 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196476 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196479 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196481 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196484 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 17:13:21.202491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196487 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196489 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196492 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196494 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196497 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196499 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196501 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196504 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196508 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196511 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196515 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196519 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196522 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196524 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196527 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196530 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196532 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196535 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196537 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 17:13:21.202996 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196540 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196542 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196545 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196548 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196550 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196552 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196555 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196558 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196560 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196568 2557 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196571 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196574 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196577 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196579 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196581 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196584 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196586 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196589 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196591 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196594 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 17:13:21.203491 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196596 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196599 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196601 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196604 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196606 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196609 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196611 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196614 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196616 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196619 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196621 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196623 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196626 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196628 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196631 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196634 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196636 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196639 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196642 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196644 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 17:13:21.203991 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196647 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196649 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196659 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196662 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196664 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196666 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:21.196670 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.196675 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.196793 2557 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.198803 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.199713 2557 server.go:1019] "Starting client certificate rotation" Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.199821 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.199860 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 17:13:21.204629 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.202714 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 17:13:21.204982 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.204170 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 17:13:21.216137 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.216116 2557 log.go:25] "Validated CRI v1 runtime API" Sep 10 17:13:21.218937 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.218918 2557 log.go:25] "Validated CRI v1 image API" Sep 10 17:13:21.219773 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.219757 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 17:13:21.222188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.222168 2557 fs.go:135] Filesystem UUIDs: map[6095aaae-5288-4181-a64a-8d0092eb2c8a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 edb8e79c-500e-499f-b931-81d33b141fa7:/dev/nvme0n1p4] Sep 10 17:13:21.222260 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.222187 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 17:13:21.227226 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.227115 2557 manager.go:216] Machine: {Timestamp:2026-09-10 17:13:21.225863677 +0000 UTC m=+0.323762028 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099285 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec265c8d87642a885f253d679939f4a5 SystemUUID:ec265c8d-8764-2a88-5f25-3d679939f4a5 BootID:811e3b75-0d0b-4327-bacc-a15934073d29 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d2:0f:5d:f2:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d2:0f:5d:f2:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:6c:37:f6:1f:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 17:13:21.227226 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.227219 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 17:13:21.227348 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.227337 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 17:13:21.230426 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.230394 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 17:13:21.230601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.230430 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 17:13:21.231619 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.231607 2557 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 17:13:21.231675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.231627 2557 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 17:13:21.231675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.231638 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 17:13:21.232420 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.232397 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 17:13:21.234359 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.234347 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 17:13:21.236899 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.236885 2557 state_mem.go:36] "Initialized new in-memory state store" Sep 10 17:13:21.237055 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.237017 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 17:13:21.237123 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.237083 2557 kubelet.go:486] "Attempting to sync node with API server" Sep 10 17:13:21.237123 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.237099 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 17:13:21.237123 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.237118 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 17:13:21.237236 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.237133 2557 kubelet.go:398] "Adding apiserver pod source" Sep 10 17:13:21.237236 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.237135 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 17:13:21.237236 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.237145 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 17:13:21.237542 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.237529 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 17:13:21.237606 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.237552 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 17:13:21.239538 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.239522 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 17:13:21.242125 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.242112 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 17:13:21.242181 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.242144 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 17:13:21.243025 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243013 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 17:13:21.243081 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243048 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 17:13:21.243081 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243056 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 17:13:21.243081 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243062 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 17:13:21.243081 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243070 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 17:13:21.243081 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243076 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 17:13:21.243212 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243089 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 17:13:21.243212 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243095 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 17:13:21.243212 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243101 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 17:13:21.243212 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243106 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 17:13:21.243212 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243114 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 17:13:21.243212 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.243120 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 17:13:21.244381 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.244368 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 17:13:21.244381 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.244381 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 17:13:21.248274 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.248254 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 17:13:21.248378 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.248338 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 17:13:21.248417 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.248389 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 17:13:21.248498 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.248488 2557 server.go:1268] "Started kubelet" Sep 10 17:13:21.251951 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.251917 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 17:13:21.252304 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.248579 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 17:13:21.252422 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.252336 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 17:13:21.252490 ip-10-0-133-191 systemd[1]: Started Kubernetes Kubelet. Sep 10 17:13:21.253850 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.253833 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 17:13:21.256862 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.256848 2557 server.go:310] "Adding debug handlers to kubelet server" Sep 10 17:13:21.257356 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.257332 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 17:13:21.259485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.259465 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 17:13:21.259663 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.259488 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 17:13:21.259663 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.259617 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:21.260251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260062 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 17:13:21.260251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260082 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 17:13:21.260251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260150 2557 reconstruct.go:97] "Volume reconstruction finished" Sep 10 17:13:21.260251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260157 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 17:13:21.260251 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.259069 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-191.ec2.internal.18d404f4c2213889 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-191.ec2.internal,UID:ip-10-0-133-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-191.ec2.internal,},FirstTimestamp:2026-09-10 17:13:21.248417929 +0000 UTC m=+0.346316282,LastTimestamp:2026-09-10 17:13:21.248417929 +0000 UTC m=+0.346316282,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-191.ec2.internal,}" Sep 10 17:13:21.260251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260159 2557 reconciler.go:29] "Reconciler: start to sync state" Sep 10 17:13:21.260568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260301 2557 factory.go:55] Registering systemd factory Sep 10 17:13:21.260568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260316 2557 factory.go:223] Registration of the systemd container factory successfully Sep 10 17:13:21.260568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260500 2557 factory.go:153] Registering CRI-O factory Sep 10 17:13:21.260568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260511 2557 factory.go:223] Registration of the crio container factory successfully Sep 10 17:13:21.260568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260548 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 17:13:21.260568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260562 2557 factory.go:103] Registering Raw factory Sep 10 17:13:21.260568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260572 2557 manager.go:1196] Started watching for new ooms in manager Sep 10 17:13:21.260990 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.260975 2557 manager.go:349] Starting recovery of all containers Sep 10 17:13:21.264615 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.264587 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 17:13:21.264753 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.264730 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 17:13:21.268984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.268909 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j9bmj" Sep 10 17:13:21.270983 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.270968 2557 manager.go:354] Recovery completed Sep 10 17:13:21.276842 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.276828 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:13:21.277316 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.277299 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j9bmj" Sep 10 17:13:21.278632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.278617 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:13:21.278704 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.278647 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:13:21.278704 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.278662 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:13:21.279339 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.279324 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 17:13:21.279339 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.279338 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 17:13:21.279456 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.279353 2557 state_mem.go:36] "Initialized new in-memory state store" Sep 10 17:13:21.279456 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.279326 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:13:21.280207 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.280192 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:13:21.280278 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.280220 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:13:21.280278 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.280245 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:13:21.280389 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.280377 2557 policy_none.go:49] "None policy: Start" Sep 10 17:13:21.280419 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.280395 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 17:13:21.280419 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.280405 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 17:13:21.281408 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.281285 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-191.ec2.internal.18d404f4c3ee3d19 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-191.ec2.internal,UID:ip-10-0-133-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-191.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-191.ec2.internal,},FirstTimestamp:2026-09-10 17:13:21.278631193 +0000 UTC m=+0.376529544,LastTimestamp:2026-09-10 17:13:21.278631193 +0000 UTC m=+0.376529544,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-191.ec2.internal,}" Sep 10 17:13:21.281650 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.281634 2557 policy_none.go:47] "Start" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.314585 2557 manager.go:337] "Starting Device Plugin manager" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.314623 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.314635 2557 server.go:87] "Starting device plugin registration server" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.314878 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.314889 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.314962 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.315083 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.315094 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.315582 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 17:13:21.318918 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.315626 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:21.378592 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.378558 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 17:13:21.379716 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.379701 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 17:13:21.379812 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.379747 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 17:13:21.379812 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.379768 2557 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 17:13:21.379901 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.379809 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 17:13:21.382317 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.382299 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:13:21.415467 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.415444 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:13:21.416189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.416175 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:13:21.416264 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.416199 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:13:21.416264 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.416210 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:13:21.416264 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.416233 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.423205 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.423183 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.423205 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.423203 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-191.ec2.internal\": node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:21.458736 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.458678 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:21.480432 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.480404 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal"] Sep 10 17:13:21.480505 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.480462 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:13:21.481691 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.481679 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:13:21.481758 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.481703 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:13:21.481758 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.481714 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:13:21.482808 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.482797 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:13:21.482940 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.482917 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.482970 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.482954 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:13:21.483490 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.483472 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:13:21.483561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.483505 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:13:21.483561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.483521 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:13:21.483561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.483482 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:13:21.483561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.483550 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:13:21.483751 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.483566 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:13:21.484556 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.484538 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.484613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.484569 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 17:13:21.485219 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.485203 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientMemory" Sep 10 17:13:21.485315 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.485228 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 17:13:21.485315 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.485248 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeHasSufficientPID" Sep 10 17:13:21.506513 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.506495 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-191.ec2.internal\" not found" node="ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.510576 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.510561 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-191.ec2.internal\" not found" node="ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.559494 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.559475 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:21.561692 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.561677 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af1fd4d122936dd15efa29d3d763213a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-191.ec2.internal\" (UID: \"af1fd4d122936dd15efa29d3d763213a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.561757 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.561699 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24c85c47d48490daee4a64b34e85f712-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal\" (UID: \"24c85c47d48490daee4a64b34e85f712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.561757 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.561716 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24c85c47d48490daee4a64b34e85f712-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal\" (UID: \"24c85c47d48490daee4a64b34e85f712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.660539 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.660515 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:21.662720 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.662704 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24c85c47d48490daee4a64b34e85f712-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal\" (UID: \"24c85c47d48490daee4a64b34e85f712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.662794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.662737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24c85c47d48490daee4a64b34e85f712-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal\" (UID: \"24c85c47d48490daee4a64b34e85f712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.662794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.662755 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af1fd4d122936dd15efa29d3d763213a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-191.ec2.internal\" (UID: \"af1fd4d122936dd15efa29d3d763213a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.662794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.662773 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24c85c47d48490daee4a64b34e85f712-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal\" (UID: \"24c85c47d48490daee4a64b34e85f712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.662904 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.662796 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af1fd4d122936dd15efa29d3d763213a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-191.ec2.internal\" (UID: \"af1fd4d122936dd15efa29d3d763213a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.662904 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.662733 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24c85c47d48490daee4a64b34e85f712-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal\" (UID: \"24c85c47d48490daee4a64b34e85f712\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.760607 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.760554 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:21.808871 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.808850 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.813449 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:21.813426 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal" Sep 10 17:13:21.860677 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.860653 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:21.961089 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:21.961052 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:22.061548 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.061482 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-191.ec2.internal\" not found" Sep 10 17:13:22.157772 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.157743 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:13:22.160073 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.160054 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" Sep 10 17:13:22.171903 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.171883 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 17:13:22.172448 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.172424 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:13:22.172983 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.172968 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal" Sep 10 17:13:22.182536 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.182521 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 17:13:22.199385 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.199365 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 17:13:22.199501 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.199485 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 17:13:22.199548 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.199527 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 17:13:22.199586 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.199527 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 17:13:22.238377 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.238349 2557 apiserver.go:52] "Watching apiserver" Sep 10 17:13:22.251041 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.251018 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 17:13:22.251405 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.251384 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-mrzvh","kube-system/konnectivity-agent-p25s2","openshift-cluster-node-tuning-operator/tuned-czxxd","openshift-dns/node-resolver-x275d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal","openshift-multus/multus-additional-cni-plugins-rljl7","openshift-multus/network-metrics-daemon-tgzqt","openshift-network-diagnostics/network-check-target-k87t9","kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j","openshift-image-registry/node-ca-zdrnp","openshift-multus/multus-6qtnb","openshift-network-operator/iptables-alerter-9ssr5","openshift-ovn-kubernetes/ovnkube-node-n64hg"] Sep 10 17:13:22.252753 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.252732 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.252836 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.252817 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:22.253723 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.253708 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:22.254730 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.254714 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.255685 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.255659 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4cdcd\"" Sep 10 17:13:22.255785 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.255689 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 17:13:22.255785 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.255711 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.255785 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.255767 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 17:13:22.256435 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.256419 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pqndg\"" Sep 10 17:13:22.256507 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.256474 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:13:22.256795 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.256777 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.256897 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.256787 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 17:13:22.257683 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.257661 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z8w2l\"" Sep 10 17:13:22.257683 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.257675 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 17:13:22.257843 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.257714 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 17:13:22.257917 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.257900 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:22.257982 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.257962 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:22.258597 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.258576 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 17:13:22.258672 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.258604 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-46qgl\"" Sep 10 17:13:22.258835 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.258820 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 17:13:22.258879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.258846 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 17:13:22.258921 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.258822 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 17:13:22.259097 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.259081 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:22.259158 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.259143 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:22.259708 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.259686 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 17:13:22.259811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.259782 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 17:13:22.260191 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.260175 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.261501 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.261483 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.261912 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.261896 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 17:13:22.262004 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.261922 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mkhpt\"" Sep 10 17:13:22.262065 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.262054 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 17:13:22.262150 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.262136 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 17:13:22.262674 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.262659 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.263348 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.263330 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 17:13:22.263420 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.263402 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 17:13:22.263518 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.263435 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bq56m\"" Sep 10 17:13:22.263573 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.263524 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 17:13:22.264189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.264166 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.264733 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.264712 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 17:13:22.265528 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265203 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-65ggg\"" Sep 10 17:13:22.265528 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265463 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-socket-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.265528 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265493 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-registration-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.265747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-device-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.265747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265579 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-sys-fs\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.265747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265618 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a38b50d7-4bd2-4605-8896-fd0d0f5b35d8-host\") pod \"node-ca-zdrnp\" (UID: \"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8\") " pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.265747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265665 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-dbus\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.265747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265696 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b7ddc91-a214-420a-8108-bd6d6ee44e88-cnibin\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.265747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265731 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6hv\" (UniqueName: \"kubernetes.io/projected/30389602-6f38-4940-ba5b-9e390e14a8f0-kube-api-access-2j6hv\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265763 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a38b50d7-4bd2-4605-8896-fd0d0f5b35d8-serviceca\") pod \"node-ca-zdrnp\" (UID: \"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8\") " pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265804 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265800 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7ddc91-a214-420a-8108-bd6d6ee44e88-system-cni-dir\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265850 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-modprobe-d\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265885 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-run\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265913 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b7ddc91-a214-420a-8108-bd6d6ee44e88-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265938 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86x9l\" (UniqueName: \"kubernetes.io/projected/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-kube-api-access-86x9l\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.265969 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b7ddc91-a214-420a-8108-bd6d6ee44e88-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266000 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:22.266047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266033 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-kubernetes\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266067 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-sysctl-d\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266110 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-sys\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266155 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-tuned\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266184 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5kht\" (UniqueName: \"kubernetes.io/projected/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-kube-api-access-t5kht\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266217 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-kubelet-config\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266272 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b7ddc91-a214-420a-8108-bd6d6ee44e88-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266303 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-systemd\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266328 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-lib-modules\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266355 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-var-lib-kubelet\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266377 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-tmp\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266406 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5ea32f-27b0-4267-98b1-9ceb809f9f44-tmp-dir\") pod \"node-resolver-x275d\" (UID: \"6e5ea32f-27b0-4267-98b1-9ceb809f9f44\") " pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266435 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-sysconfig\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e5ea32f-27b0-4267-98b1-9ceb809f9f44-hosts-file\") pod \"node-resolver-x275d\" (UID: \"6e5ea32f-27b0-4267-98b1-9ceb809f9f44\") " pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266497 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvqmd\" (UniqueName: \"kubernetes.io/projected/6e5ea32f-27b0-4267-98b1-9ceb809f9f44-kube-api-access-kvqmd\") pod \"node-resolver-x275d\" (UID: \"6e5ea32f-27b0-4267-98b1-9ceb809f9f44\") " pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266521 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:22.266614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266552 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9qmd\" (UniqueName: \"kubernetes.io/projected/6b7ddc91-a214-420a-8108-bd6d6ee44e88-kube-api-access-s9qmd\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266583 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b7ddc91-a214-420a-8108-bd6d6ee44e88-cni-binary-copy\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266649 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4010ee5-e8c9-4866-be3d-8d7b3312e4fc-agent-certs\") pod \"konnectivity-agent-p25s2\" (UID: \"c4010ee5-e8c9-4866-be3d-8d7b3312e4fc\") " pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266660 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266680 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4010ee5-e8c9-4866-be3d-8d7b3312e4fc-konnectivity-ca\") pod \"konnectivity-agent-p25s2\" (UID: \"c4010ee5-e8c9-4866-be3d-8d7b3312e4fc\") " pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266688 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266707 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-host\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266767 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266796 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jfp\" (UniqueName: \"kubernetes.io/projected/a38b50d7-4bd2-4605-8896-fd0d0f5b35d8-kube-api-access-42jfp\") pod \"node-ca-zdrnp\" (UID: \"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8\") " pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266830 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266836 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6kn7s\"" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.266853 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-sysctl-conf\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.267237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.267041 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:13:22.267749 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.267735 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 17:13:22.267969 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.267952 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 17:13:22.268046 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.267977 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 17:13:22.269451 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.269208 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 17:13:22.269918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.269896 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2vj55\"" Sep 10 17:13:22.270438 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.270411 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 17:13:22.271743 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.270410 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 17:13:22.273805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.273786 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 17:13:22.279685 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.279615 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 17:08:22 +0000 UTC" deadline="2028-06-12 09:34:55.349967736 +0000 UTC" Sep 10 17:13:22.279740 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.279688 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15376h21m33.070284697s" Sep 10 17:13:22.287542 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.287523 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:13:22.294243 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.294224 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hvwpf" Sep 10 17:13:22.303026 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.303004 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hvwpf" Sep 10 17:13:22.361648 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.361455 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 17:13:22.367027 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367001 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-sysconfig\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.367027 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367031 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e5ea32f-27b0-4267-98b1-9ceb809f9f44-hosts-file\") pod \"node-resolver-x275d\" (UID: \"6e5ea32f-27b0-4267-98b1-9ceb809f9f44\") " pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.367213 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367049 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-slash\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.367213 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367065 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-log-socket\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.367213 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367081 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-conf-dir\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.367213 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367112 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b7ddc91-a214-420a-8108-bd6d6ee44e88-cni-binary-copy\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.367213 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367137 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4010ee5-e8c9-4866-be3d-8d7b3312e4fc-agent-certs\") pod \"konnectivity-agent-p25s2\" (UID: \"c4010ee5-e8c9-4866-be3d-8d7b3312e4fc\") " pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:22.367213 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367161 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4010ee5-e8c9-4866-be3d-8d7b3312e4fc-konnectivity-ca\") pod \"konnectivity-agent-p25s2\" (UID: \"c4010ee5-e8c9-4866-be3d-8d7b3312e4fc\") " pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:22.367213 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367184 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-host\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.367213 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367200 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-run-netns\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367223 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-var-lib-openvswitch\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367238 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-system-cni-dir\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367254 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367270 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-sysctl-conf\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367289 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-device-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367316 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-sys-fs\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367341 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a38b50d7-4bd2-4605-8896-fd0d0f5b35d8-host\") pod \"node-ca-zdrnp\" (UID: \"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8\") " pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367374 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-var-lib-cni-bin\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367413 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6hv\" (UniqueName: \"kubernetes.io/projected/30389602-6f38-4940-ba5b-9e390e14a8f0-kube-api-access-2j6hv\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a38b50d7-4bd2-4605-8896-fd0d0f5b35d8-serviceca\") pod \"node-ca-zdrnp\" (UID: \"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8\") " pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367459 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-var-lib-cni-multus\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.367605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367544 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-modprobe-d\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367617 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-sys-fs\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.367662 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a38b50d7-4bd2-4605-8896-fd0d0f5b35d8-host\") pod \"node-ca-zdrnp\" (UID: \"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8\") " pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.367697 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.367793 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret podName:d2370b0a-f5f3-4ffb-a627-7a6cceae329e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:22.867766924 +0000 UTC m=+1.965665288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret") pod "global-pull-secret-syncer-mrzvh" (UID: "d2370b0a-f5f3-4ffb-a627-7a6cceae329e") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368007 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-sysconfig\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368025 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-device-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-run\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368094 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e5ea32f-27b0-4267-98b1-9ceb809f9f44-hosts-file\") pod \"node-resolver-x275d\" (UID: \"6e5ea32f-27b0-4267-98b1-9ceb809f9f44\") " pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368129 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec48f759-59e0-4feb-b663-174d64d9348e-iptables-alerter-script\") pod \"iptables-alerter-9ssr5\" (UID: \"ec48f759-59e0-4feb-b663-174d64d9348e\") " pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368133 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-sysctl-conf\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368171 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-cni-binary-copy\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.368193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368199 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-daemon-config\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368226 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b7ddc91-a214-420a-8108-bd6d6ee44e88-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368239 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-host\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368251 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-cni-bin\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368273 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-cni-dir\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368311 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-cnibin\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368336 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b7ddc91-a214-420a-8108-bd6d6ee44e88-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368358 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-run-ovn\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a570749c-795f-416e-9fb9-df195c4c80a2-ovn-node-metrics-cert\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368420 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-run-k8s-cni-cncf-io\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368443 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-run-netns\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368467 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7rbd\" (UniqueName: \"kubernetes.io/projected/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-kube-api-access-v7rbd\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368493 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-kubelet-config\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368521 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b7ddc91-a214-420a-8108-bd6d6ee44e88-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368549 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-systemd\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368575 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-var-lib-kubelet\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368563 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 17:13:22.368831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368603 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368630 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368660 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-run-multus-certs\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368687 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qmd\" (UniqueName: \"kubernetes.io/projected/6b7ddc91-a214-420a-8108-bd6d6ee44e88-kube-api-access-s9qmd\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368731 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368176 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-run\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368742 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a38b50d7-4bd2-4605-8896-fd0d0f5b35d8-serviceca\") pod \"node-ca-zdrnp\" (UID: \"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8\") " pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368789 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368865 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-kubelet-config\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368905 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4010ee5-e8c9-4866-be3d-8d7b3312e4fc-konnectivity-ca\") pod \"konnectivity-agent-p25s2\" (UID: \"c4010ee5-e8c9-4866-be3d-8d7b3312e4fc\") " pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.368939 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b7ddc91-a214-420a-8108-bd6d6ee44e88-cni-binary-copy\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369010 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-modprobe-d\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369125 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-systemd\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369196 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b7ddc91-a214-420a-8108-bd6d6ee44e88-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369200 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.369215 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369228 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jfp\" (UniqueName: \"kubernetes.io/projected/a38b50d7-4bd2-4605-8896-fd0d0f5b35d8-kube-api-access-42jfp\") pod \"node-ca-zdrnp\" (UID: \"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8\") " pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.369718 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.369285 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs podName:cb911006-5fc7-45f2-b42f-b44ccb6d0ac8 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:22.86926373 +0000 UTC m=+1.967162090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs") pod "network-metrics-daemon-tgzqt" (UID: "cb911006-5fc7-45f2-b42f-b44ccb6d0ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369311 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-etc-selinux\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369327 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6b7ddc91-a214-420a-8108-bd6d6ee44e88-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369350 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-var-lib-kubelet\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369359 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a570749c-795f-416e-9fb9-df195c4c80a2-ovnkube-config\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369480 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-etc-kubernetes\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369519 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-socket-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369579 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-registration-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369610 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-run-systemd\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369671 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-registration-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369682 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec48f759-59e0-4feb-b663-174d64d9348e-host-slash\") pod \"iptables-alerter-9ssr5\" (UID: \"ec48f759-59e0-4feb-b663-174d64d9348e\") " pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369704 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/30389602-6f38-4940-ba5b-9e390e14a8f0-socket-dir\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369720 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6b7ddc91-a214-420a-8108-bd6d6ee44e88-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369723 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-hostroot\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-dbus\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369813 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b7ddc91-a214-420a-8108-bd6d6ee44e88-cnibin\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369833 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a570749c-795f-416e-9fb9-df195c4c80a2-ovnkube-script-lib\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.370459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369849 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7hw\" (UniqueName: \"kubernetes.io/projected/ec48f759-59e0-4feb-b663-174d64d9348e-kube-api-access-hj7hw\") pod \"iptables-alerter-9ssr5\" (UID: \"ec48f759-59e0-4feb-b663-174d64d9348e\") " pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369849 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-dbus\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369866 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b7ddc91-a214-420a-8108-bd6d6ee44e88-cnibin\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369869 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-socket-dir-parent\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369911 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7ddc91-a214-420a-8108-bd6d6ee44e88-system-cni-dir\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369944 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-etc-openvswitch\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.369967 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-run-openvswitch\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370004 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b7ddc91-a214-420a-8108-bd6d6ee44e88-system-cni-dir\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370057 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-cni-netd\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370089 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86x9l\" (UniqueName: \"kubernetes.io/projected/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-kube-api-access-86x9l\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370114 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-systemd-units\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370150 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370257 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zrv\" (UniqueName: \"kubernetes.io/projected/a570749c-795f-416e-9fb9-df195c4c80a2-kube-api-access-j9zrv\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370280 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-var-lib-kubelet\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370310 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370332 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-kubernetes\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370378 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-sysctl-d\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-sys\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370473 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-tuned\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370483 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-kubernetes\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370501 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5kht\" (UniqueName: \"kubernetes.io/projected/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-kube-api-access-t5kht\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370517 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-sysctl-d\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370532 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-kubelet\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370536 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-sys\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370592 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-lib-modules\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370622 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-tmp\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370665 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5ea32f-27b0-4267-98b1-9ceb809f9f44-tmp-dir\") pod \"node-resolver-x275d\" (UID: \"6e5ea32f-27b0-4267-98b1-9ceb809f9f44\") " pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370692 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvqmd\" (UniqueName: \"kubernetes.io/projected/6e5ea32f-27b0-4267-98b1-9ceb809f9f44-kube-api-access-kvqmd\") pod \"node-resolver-x275d\" (UID: \"6e5ea32f-27b0-4267-98b1-9ceb809f9f44\") " pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370724 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-lib-modules\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370721 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-node-log\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.370794 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a570749c-795f-416e-9fb9-df195c4c80a2-env-overrides\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.371861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.371014 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e5ea32f-27b0-4267-98b1-9ceb809f9f44-tmp-dir\") pod \"node-resolver-x275d\" (UID: \"6e5ea32f-27b0-4267-98b1-9ceb809f9f44\") " pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.373337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.373163 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4010ee5-e8c9-4866-be3d-8d7b3312e4fc-agent-certs\") pod \"konnectivity-agent-p25s2\" (UID: \"c4010ee5-e8c9-4866-be3d-8d7b3312e4fc\") " pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:22.373337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.373214 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-tmp\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.373337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.373231 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-etc-tuned\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.376413 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.376317 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:13:22.376413 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.376342 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:13:22.376413 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.376361 2557 projected.go:196] Error preparing data for projected volume kube-api-access-gn6wb for pod openshift-network-diagnostics/network-check-target-k87t9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:22.376693 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.376483 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb podName:83f7f6b7-ad11-4b89-a118-b31dc14fcb8e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:22.876465449 +0000 UTC m=+1.974363791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gn6wb" (UniqueName: "kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb") pod "network-check-target-k87t9" (UID: "83f7f6b7-ad11-4b89-a118-b31dc14fcb8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:22.376877 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.376842 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6hv\" (UniqueName: \"kubernetes.io/projected/30389602-6f38-4940-ba5b-9e390e14a8f0-kube-api-access-2j6hv\") pod \"aws-ebs-csi-driver-node-5f68j\" (UID: \"30389602-6f38-4940-ba5b-9e390e14a8f0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.377699 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.377654 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jfp\" (UniqueName: \"kubernetes.io/projected/a38b50d7-4bd2-4605-8896-fd0d0f5b35d8-kube-api-access-42jfp\") pod \"node-ca-zdrnp\" (UID: \"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8\") " pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.378484 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.378455 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9qmd\" (UniqueName: \"kubernetes.io/projected/6b7ddc91-a214-420a-8108-bd6d6ee44e88-kube-api-access-s9qmd\") pod \"multus-additional-cni-plugins-rljl7\" (UID: \"6b7ddc91-a214-420a-8108-bd6d6ee44e88\") " pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.378596 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.378483 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5kht\" (UniqueName: \"kubernetes.io/projected/5fc92dde-a125-44e2-8dc0-ce5ea707db1c-kube-api-access-t5kht\") pod \"tuned-czxxd\" (UID: \"5fc92dde-a125-44e2-8dc0-ce5ea707db1c\") " pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.378596 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.378511 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86x9l\" (UniqueName: \"kubernetes.io/projected/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-kube-api-access-86x9l\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:22.379340 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.379323 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvqmd\" (UniqueName: \"kubernetes.io/projected/6e5ea32f-27b0-4267-98b1-9ceb809f9f44-kube-api-access-kvqmd\") pod \"node-resolver-x275d\" (UID: \"6e5ea32f-27b0-4267-98b1-9ceb809f9f44\") " pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.470760 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.470726 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1fd4d122936dd15efa29d3d763213a.slice/crio-5b04b91a68f38e7d6af099a96682fa83e612140ce07ef85cf47869e40530f243 WatchSource:0}: Error finding container 5b04b91a68f38e7d6af099a96682fa83e612140ce07ef85cf47869e40530f243: Status 404 returned error can't find the container with id 5b04b91a68f38e7d6af099a96682fa83e612140ce07ef85cf47869e40530f243 Sep 10 17:13:22.471995 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.471972 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-systemd-units\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472067 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472005 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472067 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472022 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zrv\" (UniqueName: \"kubernetes.io/projected/a570749c-795f-416e-9fb9-df195c4c80a2-kube-api-access-j9zrv\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472067 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472045 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-var-lib-kubelet\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.472237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472090 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-kubelet\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472111 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472147 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-systemd-units\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472164 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-node-log\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472175 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-kubelet\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472201 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a570749c-795f-416e-9fb9-df195c4c80a2-env-overrides\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472222 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-slash\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472234 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-node-log\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-log-socket\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-conf-dir\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472276 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-slash\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472296 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-log-socket\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472282 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-var-lib-kubelet\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472297 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-run-netns\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472329 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-conf-dir\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472369 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-run-netns\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472418 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-var-lib-openvswitch\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472458 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-system-cni-dir\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472472 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-var-lib-openvswitch\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472517 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-system-cni-dir\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472577 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-var-lib-cni-bin\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472617 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-var-lib-cni-multus\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.472643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472642 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec48f759-59e0-4feb-b663-174d64d9348e-iptables-alerter-script\") pod \"iptables-alerter-9ssr5\" (UID: \"ec48f759-59e0-4feb-b663-174d64d9348e\") " pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472660 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-cni-binary-copy\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472709 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-var-lib-cni-bin\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472713 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-var-lib-cni-multus\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472736 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-daemon-config\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472765 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-cni-bin\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472788 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-cni-dir\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472812 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-cnibin\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472818 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a570749c-795f-416e-9fb9-df195c4c80a2-env-overrides\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472841 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-run-ovn\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472872 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a570749c-795f-416e-9fb9-df195c4c80a2-ovn-node-metrics-cert\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472897 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-run-k8s-cni-cncf-io\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472898 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-cni-dir\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472899 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-cnibin\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472929 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-cni-bin\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472937 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-run-ovn\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472964 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-run-netns\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472977 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-run-k8s-cni-cncf-io\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.473367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.472991 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7rbd\" (UniqueName: \"kubernetes.io/projected/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-kube-api-access-v7rbd\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473026 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-run-netns\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473029 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-run-multus-certs\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473095 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a570749c-795f-416e-9fb9-df195c4c80a2-ovnkube-config\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-etc-kubernetes\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473141 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-run-systemd\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473147 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-host-run-multus-certs\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473175 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec48f759-59e0-4feb-b663-174d64d9348e-host-slash\") pod \"iptables-alerter-9ssr5\" (UID: \"ec48f759-59e0-4feb-b663-174d64d9348e\") " pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473205 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473202 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-hostroot\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473236 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec48f759-59e0-4feb-b663-174d64d9348e-iptables-alerter-script\") pod \"iptables-alerter-9ssr5\" (UID: \"ec48f759-59e0-4feb-b663-174d64d9348e\") " pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473248 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-run-systemd\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473250 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-etc-kubernetes\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473251 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a570749c-795f-416e-9fb9-df195c4c80a2-ovnkube-script-lib\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473276 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-cni-binary-copy\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473291 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec48f759-59e0-4feb-b663-174d64d9348e-host-slash\") pod \"iptables-alerter-9ssr5\" (UID: \"ec48f759-59e0-4feb-b663-174d64d9348e\") " pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.474085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7hw\" (UniqueName: \"kubernetes.io/projected/ec48f759-59e0-4feb-b663-174d64d9348e-kube-api-access-hj7hw\") pod \"iptables-alerter-9ssr5\" (UID: \"ec48f759-59e0-4feb-b663-174d64d9348e\") " pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473335 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-socket-dir-parent\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473361 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-etc-openvswitch\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-run-openvswitch\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473411 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-cni-netd\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473429 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-socket-dir-parent\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473484 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-host-cni-netd\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473514 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-run-openvswitch\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473484 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a570749c-795f-416e-9fb9-df195c4c80a2-etc-openvswitch\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473547 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-hostroot\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473704 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a570749c-795f-416e-9fb9-df195c4c80a2-ovnkube-script-lib\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.473746 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a570749c-795f-416e-9fb9-df195c4c80a2-ovnkube-config\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.474754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.474012 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-multus-daemon-config\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.476471 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.476447 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a570749c-795f-416e-9fb9-df195c4c80a2-ovn-node-metrics-cert\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.477378 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.477362 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:13:22.480396 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.480374 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c85c47d48490daee4a64b34e85f712.slice/crio-c0590574701d2cd243d1498e2606f39f736afb993f1e83429fc988b3bf7c0c6a WatchSource:0}: Error finding container c0590574701d2cd243d1498e2606f39f736afb993f1e83429fc988b3bf7c0c6a: Status 404 returned error can't find the container with id c0590574701d2cd243d1498e2606f39f736afb993f1e83429fc988b3bf7c0c6a Sep 10 17:13:22.487475 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.487446 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7rbd\" (UniqueName: \"kubernetes.io/projected/62a0c4e9-0ee9-466e-adc0-335dacce4f3b-kube-api-access-v7rbd\") pod \"multus-6qtnb\" (UID: \"62a0c4e9-0ee9-466e-adc0-335dacce4f3b\") " pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.488422 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.488400 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7hw\" (UniqueName: \"kubernetes.io/projected/ec48f759-59e0-4feb-b663-174d64d9348e-kube-api-access-hj7hw\") pod \"iptables-alerter-9ssr5\" (UID: \"ec48f759-59e0-4feb-b663-174d64d9348e\") " pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.488964 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.488944 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zrv\" (UniqueName: \"kubernetes.io/projected/a570749c-795f-416e-9fb9-df195c4c80a2-kube-api-access-j9zrv\") pod \"ovnkube-node-n64hg\" (UID: \"a570749c-795f-416e-9fb9-df195c4c80a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.581128 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.581062 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:22.590915 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.590894 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4010ee5_e8c9_4866_be3d_8d7b3312e4fc.slice/crio-321b654ad3ce005f630732ea566139b10656f5c7a24ce4aa99389b0ec05e44d1 WatchSource:0}: Error finding container 321b654ad3ce005f630732ea566139b10656f5c7a24ce4aa99389b0ec05e44d1: Status 404 returned error can't find the container with id 321b654ad3ce005f630732ea566139b10656f5c7a24ce4aa99389b0ec05e44d1 Sep 10 17:13:22.597188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.597169 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-czxxd" Sep 10 17:13:22.605202 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.605181 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc92dde_a125_44e2_8dc0_ce5ea707db1c.slice/crio-7c41f39c2bd6eb120f3f706a53d354ae04639b0e4feb8b77b85309b2aefb3125 WatchSource:0}: Error finding container 7c41f39c2bd6eb120f3f706a53d354ae04639b0e4feb8b77b85309b2aefb3125: Status 404 returned error can't find the container with id 7c41f39c2bd6eb120f3f706a53d354ae04639b0e4feb8b77b85309b2aefb3125 Sep 10 17:13:22.625126 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.625101 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x275d" Sep 10 17:13:22.633347 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.633326 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5ea32f_27b0_4267_98b1_9ceb809f9f44.slice/crio-e7760e3e4c5f821d42a504b3ef6cef741c0b6533623d6f70b87c02e6857801f4 WatchSource:0}: Error finding container e7760e3e4c5f821d42a504b3ef6cef741c0b6533623d6f70b87c02e6857801f4: Status 404 returned error can't find the container with id e7760e3e4c5f821d42a504b3ef6cef741c0b6533623d6f70b87c02e6857801f4 Sep 10 17:13:22.642199 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.642183 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rljl7" Sep 10 17:13:22.650074 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.649998 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7ddc91_a214_420a_8108_bd6d6ee44e88.slice/crio-7c05735b758964b280c03bc46791af492c2267cdfd9aaf5a6465385363c71313 WatchSource:0}: Error finding container 7c05735b758964b280c03bc46791af492c2267cdfd9aaf5a6465385363c71313: Status 404 returned error can't find the container with id 7c05735b758964b280c03bc46791af492c2267cdfd9aaf5a6465385363c71313 Sep 10 17:13:22.660652 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.660631 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" Sep 10 17:13:22.668469 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.668449 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30389602_6f38_4940_ba5b_9e390e14a8f0.slice/crio-5f035317a22240900b09465c6b387da970da83b66771fbe4d986521047c49009 WatchSource:0}: Error finding container 5f035317a22240900b09465c6b387da970da83b66771fbe4d986521047c49009: Status 404 returned error can't find the container with id 5f035317a22240900b09465c6b387da970da83b66771fbe4d986521047c49009 Sep 10 17:13:22.672015 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.671994 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zdrnp" Sep 10 17:13:22.679897 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.679876 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38b50d7_4bd2_4605_8896_fd0d0f5b35d8.slice/crio-8f341986ecd0e507b6b6336f3ffd6e7546a35b09e9870d3546190530b0b1217c WatchSource:0}: Error finding container 8f341986ecd0e507b6b6336f3ffd6e7546a35b09e9870d3546190530b0b1217c: Status 404 returned error can't find the container with id 8f341986ecd0e507b6b6336f3ffd6e7546a35b09e9870d3546190530b0b1217c Sep 10 17:13:22.697935 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.697892 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6qtnb" Sep 10 17:13:22.706462 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.706443 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a0c4e9_0ee9_466e_adc0_335dacce4f3b.slice/crio-1472a881c35c4b8dc1cdb09bab815a30f13deed3375b6211b1f42e538db03b65 WatchSource:0}: Error finding container 1472a881c35c4b8dc1cdb09bab815a30f13deed3375b6211b1f42e538db03b65: Status 404 returned error can't find the container with id 1472a881c35c4b8dc1cdb09bab815a30f13deed3375b6211b1f42e538db03b65 Sep 10 17:13:22.714972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.714955 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9ssr5" Sep 10 17:13:22.719891 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.719870 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:22.723870 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.723848 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec48f759_59e0_4feb_b663_174d64d9348e.slice/crio-e6e83a332b8574b7d2773bf0042c1b6536327e30b9c0519c0faca13b4bbcfbc2 WatchSource:0}: Error finding container e6e83a332b8574b7d2773bf0042c1b6536327e30b9c0519c0faca13b4bbcfbc2: Status 404 returned error can't find the container with id e6e83a332b8574b7d2773bf0042c1b6536327e30b9c0519c0faca13b4bbcfbc2 Sep 10 17:13:22.728406 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:22.728377 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda570749c_795f_416e_9fb9_df195c4c80a2.slice/crio-982f0a5d2dfd79dca8899c00d67ef9cc7932d0e0dc6acb44e243b0e563d485ca WatchSource:0}: Error finding container 982f0a5d2dfd79dca8899c00d67ef9cc7932d0e0dc6acb44e243b0e563d485ca: Status 404 returned error can't find the container with id 982f0a5d2dfd79dca8899c00d67ef9cc7932d0e0dc6acb44e243b0e563d485ca Sep 10 17:13:22.877193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.875669 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:22.877193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.875744 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:22.877193 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.875898 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:22.877193 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.875967 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs podName:cb911006-5fc7-45f2-b42f-b44ccb6d0ac8 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:23.875946395 +0000 UTC m=+2.973844752 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs") pod "network-metrics-daemon-tgzqt" (UID: "cb911006-5fc7-45f2-b42f-b44ccb6d0ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:22.877193 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.876540 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:22.877193 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.876603 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret podName:d2370b0a-f5f3-4ffb-a627-7a6cceae329e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:23.876586716 +0000 UTC m=+2.974485062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret") pod "global-pull-secret-syncer-mrzvh" (UID: "d2370b0a-f5f3-4ffb-a627-7a6cceae329e") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:22.976115 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:22.976059 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:22.976652 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.976251 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:13:22.976652 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.976419 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:13:22.976652 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.976434 2557 projected.go:196] Error preparing data for projected volume kube-api-access-gn6wb for pod openshift-network-diagnostics/network-check-target-k87t9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:22.976652 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:22.976516 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb podName:83f7f6b7-ad11-4b89-a118-b31dc14fcb8e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:23.976496195 +0000 UTC m=+3.074394541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gn6wb" (UniqueName: "kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb") pod "network-check-target-k87t9" (UID: "83f7f6b7-ad11-4b89-a118-b31dc14fcb8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:23.305650 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.305534 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:08:23 +0000 UTC" deadline="2028-05-03 02:22:27.407074108 +0000 UTC" Sep 10 17:13:23.305650 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.305599 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14409h9m4.101479212s" Sep 10 17:13:23.389160 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.382850 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:23.389160 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:23.382977 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:23.422686 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.422625 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9ssr5" event={"ID":"ec48f759-59e0-4feb-b663-174d64d9348e","Type":"ContainerStarted","Data":"e6e83a332b8574b7d2773bf0042c1b6536327e30b9c0519c0faca13b4bbcfbc2"} Sep 10 17:13:23.457743 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.457703 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6qtnb" event={"ID":"62a0c4e9-0ee9-466e-adc0-335dacce4f3b","Type":"ContainerStarted","Data":"1472a881c35c4b8dc1cdb09bab815a30f13deed3375b6211b1f42e538db03b65"} Sep 10 17:13:23.465789 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.465735 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zdrnp" event={"ID":"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8","Type":"ContainerStarted","Data":"8f341986ecd0e507b6b6336f3ffd6e7546a35b09e9870d3546190530b0b1217c"} Sep 10 17:13:23.489389 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.489355 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" event={"ID":"30389602-6f38-4940-ba5b-9e390e14a8f0","Type":"ContainerStarted","Data":"5f035317a22240900b09465c6b387da970da83b66771fbe4d986521047c49009"} Sep 10 17:13:23.500786 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.500759 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljl7" event={"ID":"6b7ddc91-a214-420a-8108-bd6d6ee44e88","Type":"ContainerStarted","Data":"7c05735b758964b280c03bc46791af492c2267cdfd9aaf5a6465385363c71313"} Sep 10 17:13:23.512385 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.512357 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x275d" event={"ID":"6e5ea32f-27b0-4267-98b1-9ceb809f9f44","Type":"ContainerStarted","Data":"e7760e3e4c5f821d42a504b3ef6cef741c0b6533623d6f70b87c02e6857801f4"} Sep 10 17:13:23.519773 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.519744 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-czxxd" event={"ID":"5fc92dde-a125-44e2-8dc0-ce5ea707db1c","Type":"ContainerStarted","Data":"7c41f39c2bd6eb120f3f706a53d354ae04639b0e4feb8b77b85309b2aefb3125"} Sep 10 17:13:23.537309 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.537286 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal" event={"ID":"af1fd4d122936dd15efa29d3d763213a","Type":"ContainerStarted","Data":"5b04b91a68f38e7d6af099a96682fa83e612140ce07ef85cf47869e40530f243"} Sep 10 17:13:23.548363 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.548337 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerStarted","Data":"982f0a5d2dfd79dca8899c00d67ef9cc7932d0e0dc6acb44e243b0e563d485ca"} Sep 10 17:13:23.568914 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.568841 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p25s2" event={"ID":"c4010ee5-e8c9-4866-be3d-8d7b3312e4fc","Type":"ContainerStarted","Data":"321b654ad3ce005f630732ea566139b10656f5c7a24ce4aa99389b0ec05e44d1"} Sep 10 17:13:23.577724 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.577701 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" event={"ID":"24c85c47d48490daee4a64b34e85f712","Type":"ContainerStarted","Data":"c0590574701d2cd243d1498e2606f39f736afb993f1e83429fc988b3bf7c0c6a"} Sep 10 17:13:23.678905 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.678876 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:13:23.841727 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.841646 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:13:23.883688 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.883655 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:23.883859 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.883740 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:23.883927 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:23.883865 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:23.883927 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:23.883889 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:23.884026 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:23.883960 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret podName:d2370b0a-f5f3-4ffb-a627-7a6cceae329e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:25.883937563 +0000 UTC m=+4.981835907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret") pod "global-pull-secret-syncer-mrzvh" (UID: "d2370b0a-f5f3-4ffb-a627-7a6cceae329e") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:23.884026 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:23.883982 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs podName:cb911006-5fc7-45f2-b42f-b44ccb6d0ac8 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:25.883971045 +0000 UTC m=+4.981869390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs") pod "network-metrics-daemon-tgzqt" (UID: "cb911006-5fc7-45f2-b42f-b44ccb6d0ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:23.984130 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:23.984086 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:23.984311 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:23.984273 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:13:23.984311 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:23.984294 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:13:23.984311 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:23.984306 2557 projected.go:196] Error preparing data for projected volume kube-api-access-gn6wb for pod openshift-network-diagnostics/network-check-target-k87t9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:23.984480 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:23.984360 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb podName:83f7f6b7-ad11-4b89-a118-b31dc14fcb8e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:25.984341485 +0000 UTC m=+5.082239831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gn6wb" (UniqueName: "kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb") pod "network-check-target-k87t9" (UID: "83f7f6b7-ad11-4b89-a118-b31dc14fcb8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:24.306617 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:24.306516 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:08:23 +0000 UTC" deadline="2028-04-29 10:33:03.763681489 +0000 UTC" Sep 10 17:13:24.306617 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:24.306556 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14321h19m39.45713005s" Sep 10 17:13:24.381529 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:24.381487 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:24.381716 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:24.381647 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:24.381716 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:24.381694 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:24.381890 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:24.381812 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:24.900878 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:24.900843 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 17:13:25.382148 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:25.381551 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:25.382148 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:25.381702 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:25.902147 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:25.902111 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:25.902337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:25.902208 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:25.902402 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:25.902346 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:25.902463 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:25.902408 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret podName:d2370b0a-f5f3-4ffb-a627-7a6cceae329e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:29.902388319 +0000 UTC m=+9.000286659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret") pod "global-pull-secret-syncer-mrzvh" (UID: "d2370b0a-f5f3-4ffb-a627-7a6cceae329e") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:25.902842 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:25.902821 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:25.902962 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:25.902880 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs podName:cb911006-5fc7-45f2-b42f-b44ccb6d0ac8 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:29.902861928 +0000 UTC m=+9.000760274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs") pod "network-metrics-daemon-tgzqt" (UID: "cb911006-5fc7-45f2-b42f-b44ccb6d0ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:26.003363 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:26.003329 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:26.003512 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:26.003498 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:13:26.003581 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:26.003523 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:13:26.003581 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:26.003536 2557 projected.go:196] Error preparing data for projected volume kube-api-access-gn6wb for pod openshift-network-diagnostics/network-check-target-k87t9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:26.003692 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:26.003597 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb podName:83f7f6b7-ad11-4b89-a118-b31dc14fcb8e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:30.003579041 +0000 UTC m=+9.101477384 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gn6wb" (UniqueName: "kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb") pod "network-check-target-k87t9" (UID: "83f7f6b7-ad11-4b89-a118-b31dc14fcb8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:26.380946 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:26.380854 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:26.380946 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:26.380896 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:26.381202 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:26.381004 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:26.381202 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:26.381156 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:27.380726 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:27.380692 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:27.381412 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:27.380851 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:28.380748 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:28.380704 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:28.380748 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:28.380741 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:28.381314 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:28.380875 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:28.381362 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:28.381310 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:29.380895 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:29.380676 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:29.380895 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:29.380824 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:29.941540 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:29.941497 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:29.941745 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:29.941566 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:29.941745 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:29.941689 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:29.941745 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:29.941692 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:29.941927 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:29.941756 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs podName:cb911006-5fc7-45f2-b42f-b44ccb6d0ac8 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:37.94173608 +0000 UTC m=+17.039634421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs") pod "network-metrics-daemon-tgzqt" (UID: "cb911006-5fc7-45f2-b42f-b44ccb6d0ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:29.941927 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:29.941774 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret podName:d2370b0a-f5f3-4ffb-a627-7a6cceae329e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:37.941766251 +0000 UTC m=+17.039664593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret") pod "global-pull-secret-syncer-mrzvh" (UID: "d2370b0a-f5f3-4ffb-a627-7a6cceae329e") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:30.042910 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:30.042876 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:30.043628 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:30.043107 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:13:30.043628 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:30.043137 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:13:30.043628 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:30.043152 2557 projected.go:196] Error preparing data for projected volume kube-api-access-gn6wb for pod openshift-network-diagnostics/network-check-target-k87t9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:30.043628 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:30.043224 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb podName:83f7f6b7-ad11-4b89-a118-b31dc14fcb8e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:38.043202977 +0000 UTC m=+17.141101345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gn6wb" (UniqueName: "kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb") pod "network-check-target-k87t9" (UID: "83f7f6b7-ad11-4b89-a118-b31dc14fcb8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:30.380496 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:30.380413 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:30.380655 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:30.380561 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:30.380655 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:30.380605 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:30.380776 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:30.380689 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:31.382290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:31.381760 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:31.382290 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:31.381883 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:32.380245 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:32.380209 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:32.380425 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:32.380348 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:32.380425 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:32.380393 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:32.380546 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:32.380522 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:33.381101 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:33.381066 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:33.381641 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:33.381208 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:34.380796 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:34.380766 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:34.380898 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:34.380870 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:34.380965 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:34.380934 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:34.381073 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:34.381055 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:35.383044 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:35.383016 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:35.383436 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:35.383137 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:36.380526 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:36.380493 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:36.380695 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:36.380534 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:36.380695 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:36.380633 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:36.380795 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:36.380749 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:37.380818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:37.380782 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:37.381298 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:37.380918 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:38.002715 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:38.002691 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:38.002796 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:38.002746 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:38.002843 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.002827 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:38.002843 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.002828 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:38.002906 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.002869 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret podName:d2370b0a-f5f3-4ffb-a627-7a6cceae329e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:54.002856488 +0000 UTC m=+33.100754826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret") pod "global-pull-secret-syncer-mrzvh" (UID: "d2370b0a-f5f3-4ffb-a627-7a6cceae329e") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:38.002906 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.002882 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs podName:cb911006-5fc7-45f2-b42f-b44ccb6d0ac8 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:54.002875579 +0000 UTC m=+33.100773918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs") pod "network-metrics-daemon-tgzqt" (UID: "cb911006-5fc7-45f2-b42f-b44ccb6d0ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:38.103384 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:38.103341 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:38.103475 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.103461 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:13:38.103509 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.103476 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:13:38.103509 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.103484 2557 projected.go:196] Error preparing data for projected volume kube-api-access-gn6wb for pod openshift-network-diagnostics/network-check-target-k87t9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:38.103573 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.103523 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb podName:83f7f6b7-ad11-4b89-a118-b31dc14fcb8e nodeName:}" failed. No retries permitted until 2026-09-10 17:13:54.103510127 +0000 UTC m=+33.201408470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gn6wb" (UniqueName: "kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb") pod "network-check-target-k87t9" (UID: "83f7f6b7-ad11-4b89-a118-b31dc14fcb8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:38.380432 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:38.380360 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:38.380573 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.380481 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:38.380573 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:38.380546 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:38.380702 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:38.380664 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:39.382633 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:39.382607 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:39.383117 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:39.382720 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:40.379971 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:40.379934 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:40.380180 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:40.379986 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:40.380180 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:40.380080 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:40.380291 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:40.380210 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:41.382797 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:41.382767 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:41.383193 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:41.382864 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:42.380620 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.380285 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:42.380796 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.380481 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:42.380851 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:42.380737 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:42.380889 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:42.380828 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:42.620098 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.619838 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-czxxd" event={"ID":"5fc92dde-a125-44e2-8dc0-ce5ea707db1c","Type":"ContainerStarted","Data":"d29e999dcd272d54c447515ca201c6320b4a3434dc4ab2d329f520f3c5426a29"} Sep 10 17:13:42.622412 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.622351 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal" event={"ID":"af1fd4d122936dd15efa29d3d763213a","Type":"ContainerStarted","Data":"4d64bf35bd7cbb72c24146662faf43a72c11e4085d38297abd3bcb47e2f491ac"} Sep 10 17:13:42.629241 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.629217 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:13:42.629687 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.629650 2557 generic.go:356] "Generic (PLEG): container finished" podID="a570749c-795f-416e-9fb9-df195c4c80a2" containerID="5faa5bd907ff8492bba736b761335acda6e9251518679cae76621ccbb96fd29e" exitCode=1 Sep 10 17:13:42.629776 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.629694 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerStarted","Data":"3ae0e5acd1d7c3de0577ddf6f8762721093eac807797cb4b6770542ba33a9767"} Sep 10 17:13:42.629896 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.629873 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerStarted","Data":"0e06df9dc0ad76067a37427078c0d7c1b346ebdbf181fb78803d4eed3cc436eb"} Sep 10 17:13:42.629959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.629949 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerDied","Data":"5faa5bd907ff8492bba736b761335acda6e9251518679cae76621ccbb96fd29e"} Sep 10 17:13:42.630009 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.630001 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerStarted","Data":"2a107c8532af583312854068ccba7676ea91becdd30399e8d40bbd8e7bc59b45"} Sep 10 17:13:42.631364 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.631314 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p25s2" event={"ID":"c4010ee5-e8c9-4866-be3d-8d7b3312e4fc","Type":"ContainerStarted","Data":"651a4097d85433969dd3c69813261909e75cda978a053d4d0bd05deb0b1455e5"} Sep 10 17:13:42.633597 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.633570 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6qtnb" event={"ID":"62a0c4e9-0ee9-466e-adc0-335dacce4f3b","Type":"ContainerStarted","Data":"cb0cf67cbecf7eb78f58b3356035d9f44ce86f79b0a54e3a06182652f69d4bbf"} Sep 10 17:13:42.637400 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.637350 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-czxxd" podStartSLOduration=1.414953522 podStartE2EDuration="20.637332508s" totalImagesPullingTime="19.222378986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:42.636331888 +0000 UTC m=+21.734230250" watchObservedRunningTime="2026-09-10 17:13:42.637332508 +0000 UTC m=+21.735230870" Sep 10 17:13:42.649916 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.649861 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-191.ec2.internal" podStartSLOduration=20.649842783 podStartE2EDuration="20.649842783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:42.649146008 +0000 UTC m=+21.747044371" watchObservedRunningTime="2026-09-10 17:13:42.649842783 +0000 UTC m=+21.747741143" Sep 10 17:13:42.664840 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.664802 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p25s2" podStartSLOduration=1.430463337 podStartE2EDuration="20.664789792s" totalImagesPullingTime="19.234326455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:42.664497038 +0000 UTC m=+21.762395401" watchObservedRunningTime="2026-09-10 17:13:42.664789792 +0000 UTC m=+21.762688152" Sep 10 17:13:42.681524 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:42.681486 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6qtnb" podStartSLOduration=1.26725139 podStartE2EDuration="20.681474324s" totalImagesPullingTime="19.414222934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:42.68142678 +0000 UTC m=+21.779325140" watchObservedRunningTime="2026-09-10 17:13:42.681474324 +0000 UTC m=+21.779372685" Sep 10 17:13:43.380984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.380950 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:43.381166 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:43.381079 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:43.636725 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.636648 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zdrnp" event={"ID":"a38b50d7-4bd2-4605-8896-fd0d0f5b35d8","Type":"ContainerStarted","Data":"751b487c883209598a3c7b198b9a0c621877b8060994a01a00996280e073f216"} Sep 10 17:13:43.638132 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.638105 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" event={"ID":"30389602-6f38-4940-ba5b-9e390e14a8f0","Type":"ContainerStarted","Data":"d91f523999b28dc49bc9572de93220966f99d8c7433cfc9a93e5d7f4fb614e78"} Sep 10 17:13:43.639572 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.639546 2557 generic.go:356] "Generic (PLEG): container finished" podID="6b7ddc91-a214-420a-8108-bd6d6ee44e88" containerID="1de53b0083049e2b830ff9f49da11b059ef9ea62ec25484d0a327fd59e31324f" exitCode=0 Sep 10 17:13:43.639665 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.639632 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljl7" event={"ID":"6b7ddc91-a214-420a-8108-bd6d6ee44e88","Type":"ContainerDied","Data":"1de53b0083049e2b830ff9f49da11b059ef9ea62ec25484d0a327fd59e31324f"} Sep 10 17:13:43.641152 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.641130 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x275d" event={"ID":"6e5ea32f-27b0-4267-98b1-9ceb809f9f44","Type":"ContainerStarted","Data":"3f1bcce68c20f063d7e1ea6541dc74292b9f263c776359922eb48fe2386ccfbc"} Sep 10 17:13:43.644064 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.644046 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:13:43.644504 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.644481 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerStarted","Data":"076b3503c7f6694e6be19bfd72c00f990b2796ea0d34b6d9add33031bae5ebe1"} Sep 10 17:13:43.644589 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.644560 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerStarted","Data":"02ee086192e2c2af8a5ab33bf642700dc87ee4f391d1be793bcefee511a84cb4"} Sep 10 17:13:43.645913 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.645891 2557 generic.go:356] "Generic (PLEG): container finished" podID="24c85c47d48490daee4a64b34e85f712" containerID="5b5a5cc232b81cd9fee3ee05f01847fc947baf3e7903778ba6202c255137abbc" exitCode=0 Sep 10 17:13:43.645979 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.645966 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" event={"ID":"24c85c47d48490daee4a64b34e85f712","Type":"ContainerDied","Data":"5b5a5cc232b81cd9fee3ee05f01847fc947baf3e7903778ba6202c255137abbc"} Sep 10 17:13:43.647891 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.647861 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9ssr5" event={"ID":"ec48f759-59e0-4feb-b663-174d64d9348e","Type":"ContainerStarted","Data":"57fd0e9f11d07fbf336e10daf7d77aa0dfd1aeb07ec7b67e856f6cfbd82ded54"} Sep 10 17:13:43.676275 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.676239 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zdrnp" podStartSLOduration=2.531284531 podStartE2EDuration="21.676226122s" totalImagesPullingTime="19.144941591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:43.656725456 +0000 UTC m=+22.754623821" watchObservedRunningTime="2026-09-10 17:13:43.676226122 +0000 UTC m=+22.774124527" Sep 10 17:13:43.682184 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.682164 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:43.682894 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.682876 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:43.715825 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.713818 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x275d" podStartSLOduration=2.522383272 podStartE2EDuration="21.713801223s" totalImagesPullingTime="19.191417951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:43.690996999 +0000 UTC m=+22.788895374" watchObservedRunningTime="2026-09-10 17:13:43.713801223 +0000 UTC m=+22.811699585" Sep 10 17:13:43.729613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.729455 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9ssr5" podStartSLOduration=2.628242693 podStartE2EDuration="21.729443307s" totalImagesPullingTime="19.101200614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:43.72928751 +0000 UTC m=+22.827185870" watchObservedRunningTime="2026-09-10 17:13:43.729443307 +0000 UTC m=+22.827341667" Sep 10 17:13:43.946526 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:43.946505 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 17:13:44.330218 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.330086 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T17:13:43.946521458Z","UUID":"489ccf3c-a948-49d5-a219-94c036f6e17c","Handler":null,"Name":"","Endpoint":""} Sep 10 17:13:44.331688 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.331650 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 17:13:44.331688 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.331687 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 17:13:44.379992 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.379964 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:44.380127 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:44.380096 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:44.380201 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.380148 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:44.380260 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:44.380226 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:44.653433 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.653356 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" event={"ID":"24c85c47d48490daee4a64b34e85f712","Type":"ContainerStarted","Data":"909e6cccde51f80129cd69f4ef2924c53f257fb8a7e020407d70ec2842dbecea"} Sep 10 17:13:44.655703 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.655674 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" event={"ID":"30389602-6f38-4940-ba5b-9e390e14a8f0","Type":"ContainerStarted","Data":"1fc39ba86b4814e17cf84183ee1edad41f5323544b4ce0685f0e8a75b677c1b4"} Sep 10 17:13:44.656112 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.656096 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:44.656653 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.656628 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p25s2" Sep 10 17:13:44.684182 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:44.684147 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-191.ec2.internal" podStartSLOduration=22.684130519 podStartE2EDuration="22.684130519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:44.66818129 +0000 UTC m=+23.766079653" watchObservedRunningTime="2026-09-10 17:13:44.684130519 +0000 UTC m=+23.782028886" Sep 10 17:13:45.381044 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:45.380994 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:45.381140 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:45.381120 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:45.661923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:45.661810 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" event={"ID":"30389602-6f38-4940-ba5b-9e390e14a8f0","Type":"ContainerStarted","Data":"bbb5fe6bd5dc51a896dde1370e9247bf61aad1b41c16073e7ea991a4213b12fe"} Sep 10 17:13:45.665343 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:45.665319 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:13:45.665782 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:45.665751 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerStarted","Data":"d58ac255f0e34454137e2262b3e48df3384b77245f2ba9d52500cd1b01d53f1b"} Sep 10 17:13:45.680942 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:45.680893 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5f68j" podStartSLOduration=2.001128716 podStartE2EDuration="23.680879116s" totalImagesPullingTime="21.6797504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:45.680734911 +0000 UTC m=+24.778633273" watchObservedRunningTime="2026-09-10 17:13:45.680879116 +0000 UTC m=+24.778777477" Sep 10 17:13:46.380574 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:46.380545 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:46.380738 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:46.380665 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:46.380738 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:46.380729 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:46.380886 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:46.380860 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:47.383939 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:47.383914 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:47.384349 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:47.384025 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:47.674100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:47.674076 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:13:48.380632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.380554 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:48.380764 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.380557 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:48.380764 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:48.380654 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:48.380835 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:48.380762 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:48.677906 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.677840 2557 generic.go:356] "Generic (PLEG): container finished" podID="6b7ddc91-a214-420a-8108-bd6d6ee44e88" containerID="193d0bf03b55b9fdc963ec4fe48001289f9bfdd35616e3e937bf490e1d34a4f9" exitCode=0 Sep 10 17:13:48.678477 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.677934 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljl7" event={"ID":"6b7ddc91-a214-420a-8108-bd6d6ee44e88","Type":"ContainerDied","Data":"193d0bf03b55b9fdc963ec4fe48001289f9bfdd35616e3e937bf490e1d34a4f9"} Sep 10 17:13:48.684395 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.682404 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:13:48.684395 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.682999 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerStarted","Data":"ab52bae544f55607a2bfe08e59592c08585db8fd74c31164594305729ccfba2a"} Sep 10 17:13:48.684395 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.683827 2557 scope.go:117] "RemoveContainer" containerID="5faa5bd907ff8492bba736b761335acda6e9251518679cae76621ccbb96fd29e" Sep 10 17:13:48.684598 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.684424 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:48.684598 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.684516 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:48.684598 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.684580 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:48.699818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.699798 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:48.701257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:48.701239 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:13:49.380536 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.380401 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:49.380637 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:49.380619 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:49.577450 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.577361 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mrzvh"] Sep 10 17:13:49.580614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.580592 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k87t9"] Sep 10 17:13:49.580728 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.580688 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:49.580831 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:49.580766 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:49.581337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.581314 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tgzqt"] Sep 10 17:13:49.581434 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.581421 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:49.581527 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:49.581508 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:49.686771 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.686741 2557 generic.go:356] "Generic (PLEG): container finished" podID="6b7ddc91-a214-420a-8108-bd6d6ee44e88" containerID="449d75cca61bb71f8ea49cf76918ae68b24c1333792d3fd1d9615951edc638b8" exitCode=0 Sep 10 17:13:49.687302 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.686822 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljl7" event={"ID":"6b7ddc91-a214-420a-8108-bd6d6ee44e88","Type":"ContainerDied","Data":"449d75cca61bb71f8ea49cf76918ae68b24c1333792d3fd1d9615951edc638b8"} Sep 10 17:13:49.691001 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.690984 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:13:49.691363 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.691347 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:49.691466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.691388 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" event={"ID":"a570749c-795f-416e-9fb9-df195c4c80a2","Type":"ContainerStarted","Data":"525f439643d05c259e73d6fe55960aa8bd86eb7bc6230deec1f5c3fd53c23dc0"} Sep 10 17:13:49.691466 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:49.691445 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:49.733972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:49.733939 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" podStartSLOduration=8.601482057 podStartE2EDuration="27.733928781s" totalImagesPullingTime="19.132446724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:49.733807426 +0000 UTC m=+28.831705787" watchObservedRunningTime="2026-09-10 17:13:49.733928781 +0000 UTC m=+28.831827141" Sep 10 17:13:50.695952 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:50.695888 2557 generic.go:356] "Generic (PLEG): container finished" podID="6b7ddc91-a214-420a-8108-bd6d6ee44e88" containerID="02c61e253f45424ef1295c90fb6f0d64265d11d713dd7a32ea81eb9e9c8adaab" exitCode=0 Sep 10 17:13:50.696270 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:50.695961 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljl7" event={"ID":"6b7ddc91-a214-420a-8108-bd6d6ee44e88","Type":"ContainerDied","Data":"02c61e253f45424ef1295c90fb6f0d64265d11d713dd7a32ea81eb9e9c8adaab"} Sep 10 17:13:51.385044 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:51.385001 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:51.385044 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:51.385130 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:51.385044 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:51.385192 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:51.385044 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:51.385318 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:51.385044 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:51.385616 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:51.385044 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:51.385686 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:53.380542 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:53.380464 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:53.380956 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:53.380464 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:53.380956 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:53.380595 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:53.380956 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:53.380464 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:53.380956 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:53.380660 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:53.380956 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:53.380708 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:54.021831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:54.021798 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:54.021985 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:54.021886 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:54.021985 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:54.021936 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:54.022108 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:54.022007 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs podName:cb911006-5fc7-45f2-b42f-b44ccb6d0ac8 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:26.021985996 +0000 UTC m=+65.119884336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs") pod "network-metrics-daemon-tgzqt" (UID: "cb911006-5fc7-45f2-b42f-b44ccb6d0ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 17:13:54.022108 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:54.022007 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:54.022108 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:54.022060 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret podName:d2370b0a-f5f3-4ffb-a627-7a6cceae329e nodeName:}" failed. No retries permitted until 2026-09-10 17:14:26.022050027 +0000 UTC m=+65.119948369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret") pod "global-pull-secret-syncer-mrzvh" (UID: "d2370b0a-f5f3-4ffb-a627-7a6cceae329e") : object "kube-system"/"original-pull-secret" not registered Sep 10 17:13:54.122423 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:54.122345 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:54.122556 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:54.122516 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 17:13:54.122556 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:54.122535 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 17:13:54.122556 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:54.122545 2557 projected.go:196] Error preparing data for projected volume kube-api-access-gn6wb for pod openshift-network-diagnostics/network-check-target-k87t9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:54.122709 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:54.122592 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb podName:83f7f6b7-ad11-4b89-a118-b31dc14fcb8e nodeName:}" failed. No retries permitted until 2026-09-10 17:14:26.122578764 +0000 UTC m=+65.220477116 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gn6wb" (UniqueName: "kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb") pod "network-check-target-k87t9" (UID: "83f7f6b7-ad11-4b89-a118-b31dc14fcb8e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 17:13:55.380078 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:55.380009 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:55.380410 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:55.380009 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:55.380410 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:55.380138 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:55.380410 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:55.380130 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k87t9" podUID="83f7f6b7-ad11-4b89-a118-b31dc14fcb8e" Sep 10 17:13:55.380410 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:55.380211 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mrzvh" podUID="d2370b0a-f5f3-4ffb-a627-7a6cceae329e" Sep 10 17:13:55.380410 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:55.380293 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzqt" podUID="cb911006-5fc7-45f2-b42f-b44ccb6d0ac8" Sep 10 17:13:55.710872 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:55.710800 2557 generic.go:356] "Generic (PLEG): container finished" podID="6b7ddc91-a214-420a-8108-bd6d6ee44e88" containerID="9adb17b18c80ac55e1d1530828dd43676fefb4ceab7b36130ce94af42e98b946" exitCode=0 Sep 10 17:13:55.710872 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:55.710852 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljl7" event={"ID":"6b7ddc91-a214-420a-8108-bd6d6ee44e88","Type":"ContainerDied","Data":"9adb17b18c80ac55e1d1530828dd43676fefb4ceab7b36130ce94af42e98b946"} Sep 10 17:13:56.135186 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.134932 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-191.ec2.internal" event="NodeReady" Sep 10 17:13:56.135307 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.135239 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 17:13:56.181439 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.181410 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lsk5m"] Sep 10 17:13:56.195332 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.195310 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9549r"] Sep 10 17:13:56.195439 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.195421 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.197777 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.197757 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 17:13:56.198104 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.198087 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2rm4x\"" Sep 10 17:13:56.198216 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.198123 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 17:13:56.209215 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.209197 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lsk5m"] Sep 10 17:13:56.209299 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.209226 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9549r"] Sep 10 17:13:56.209299 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.209286 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9549r" Sep 10 17:13:56.211548 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.211530 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 17:13:56.211644 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.211598 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 17:13:56.211644 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.211630 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nn99k\"" Sep 10 17:13:56.211747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.211631 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 17:13:56.338505 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.338481 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c81cc54e-dfe7-45a1-a04c-b9abba50450d-config-volume\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.338613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.338528 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81cc54e-dfe7-45a1-a04c-b9abba50450d-tmp-dir\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.338613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.338553 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8bd1e76-8725-4094-a2f5-b4d94d75abf7-cert\") pod \"ingress-canary-9549r\" (UID: \"e8bd1e76-8725-4094-a2f5-b4d94d75abf7\") " pod="openshift-ingress-canary/ingress-canary-9549r" Sep 10 17:13:56.338613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.338596 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.338765 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.338630 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6l8k\" (UniqueName: \"kubernetes.io/projected/c81cc54e-dfe7-45a1-a04c-b9abba50450d-kube-api-access-n6l8k\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.338765 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.338698 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qxzk\" (UniqueName: \"kubernetes.io/projected/e8bd1e76-8725-4094-a2f5-b4d94d75abf7-kube-api-access-6qxzk\") pod \"ingress-canary-9549r\" (UID: \"e8bd1e76-8725-4094-a2f5-b4d94d75abf7\") " pod="openshift-ingress-canary/ingress-canary-9549r" Sep 10 17:13:56.439118 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.439057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6l8k\" (UniqueName: \"kubernetes.io/projected/c81cc54e-dfe7-45a1-a04c-b9abba50450d-kube-api-access-n6l8k\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.439118 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.439099 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qxzk\" (UniqueName: \"kubernetes.io/projected/e8bd1e76-8725-4094-a2f5-b4d94d75abf7-kube-api-access-6qxzk\") pod \"ingress-canary-9549r\" (UID: \"e8bd1e76-8725-4094-a2f5-b4d94d75abf7\") " pod="openshift-ingress-canary/ingress-canary-9549r" Sep 10 17:13:56.439452 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.439127 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c81cc54e-dfe7-45a1-a04c-b9abba50450d-config-volume\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.439452 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.439316 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81cc54e-dfe7-45a1-a04c-b9abba50450d-tmp-dir\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.439452 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.439352 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8bd1e76-8725-4094-a2f5-b4d94d75abf7-cert\") pod \"ingress-canary-9549r\" (UID: \"e8bd1e76-8725-4094-a2f5-b4d94d75abf7\") " pod="openshift-ingress-canary/ingress-canary-9549r" Sep 10 17:13:56.439452 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.439370 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.439452 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:56.439450 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:13:56.439694 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:56.439497 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls podName:c81cc54e-dfe7-45a1-a04c-b9abba50450d nodeName:}" failed. No retries permitted until 2026-09-10 17:13:56.939485058 +0000 UTC m=+36.037383409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls") pod "dns-default-lsk5m" (UID: "c81cc54e-dfe7-45a1-a04c-b9abba50450d") : secret "dns-default-metrics-tls" not found Sep 10 17:13:56.439694 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.439674 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c81cc54e-dfe7-45a1-a04c-b9abba50450d-config-volume\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.439694 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.439676 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81cc54e-dfe7-45a1-a04c-b9abba50450d-tmp-dir\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.443231 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.443205 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8bd1e76-8725-4094-a2f5-b4d94d75abf7-cert\") pod \"ingress-canary-9549r\" (UID: \"e8bd1e76-8725-4094-a2f5-b4d94d75abf7\") " pod="openshift-ingress-canary/ingress-canary-9549r" Sep 10 17:13:56.447447 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.447426 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qxzk\" (UniqueName: \"kubernetes.io/projected/e8bd1e76-8725-4094-a2f5-b4d94d75abf7-kube-api-access-6qxzk\") pod \"ingress-canary-9549r\" (UID: \"e8bd1e76-8725-4094-a2f5-b4d94d75abf7\") " pod="openshift-ingress-canary/ingress-canary-9549r" Sep 10 17:13:56.447522 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.447473 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6l8k\" (UniqueName: \"kubernetes.io/projected/c81cc54e-dfe7-45a1-a04c-b9abba50450d-kube-api-access-n6l8k\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.519550 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.519529 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9549r" Sep 10 17:13:56.716553 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.716485 2557 generic.go:356] "Generic (PLEG): container finished" podID="6b7ddc91-a214-420a-8108-bd6d6ee44e88" containerID="b267b13dcce9f31adbf3e63085f8089826d93ff14755508a133b30feeda68903" exitCode=0 Sep 10 17:13:56.716553 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.716541 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljl7" event={"ID":"6b7ddc91-a214-420a-8108-bd6d6ee44e88","Type":"ContainerDied","Data":"b267b13dcce9f31adbf3e63085f8089826d93ff14755508a133b30feeda68903"} Sep 10 17:13:56.810869 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.810846 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9549r"] Sep 10 17:13:56.814084 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:13:56.814061 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bd1e76_8725_4094_a2f5_b4d94d75abf7.slice/crio-3a345bbd1b9d7dce785d836cdfa34392dee454196ddae354e0e1479f97ebca5c WatchSource:0}: Error finding container 3a345bbd1b9d7dce785d836cdfa34392dee454196ddae354e0e1479f97ebca5c: Status 404 returned error can't find the container with id 3a345bbd1b9d7dce785d836cdfa34392dee454196ddae354e0e1479f97ebca5c Sep 10 17:13:56.943711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:56.943690 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:56.943827 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:56.943812 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:13:56.943881 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:56.943861 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls podName:c81cc54e-dfe7-45a1-a04c-b9abba50450d nodeName:}" failed. No retries permitted until 2026-09-10 17:13:57.943845583 +0000 UTC m=+37.041743933 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls") pod "dns-default-lsk5m" (UID: "c81cc54e-dfe7-45a1-a04c-b9abba50450d") : secret "dns-default-metrics-tls" not found Sep 10 17:13:57.380510 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.380481 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:13:57.380667 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.380481 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:13:57.380736 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.380481 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:13:57.383864 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.383720 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g7xkh\"" Sep 10 17:13:57.383864 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.383756 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 17:13:57.383864 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.383779 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n4rqr\"" Sep 10 17:13:57.383864 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.383782 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 17:13:57.384316 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.384059 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 17:13:57.384316 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.384235 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 17:13:57.721057 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.720969 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9549r" event={"ID":"e8bd1e76-8725-4094-a2f5-b4d94d75abf7","Type":"ContainerStarted","Data":"3a345bbd1b9d7dce785d836cdfa34392dee454196ddae354e0e1479f97ebca5c"} Sep 10 17:13:57.725172 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.725144 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rljl7" event={"ID":"6b7ddc91-a214-420a-8108-bd6d6ee44e88","Type":"ContainerStarted","Data":"9deea96520861bd94bc1d1479ccffed2b02a323b84abe49eed218f368e9b8ea0"} Sep 10 17:13:57.748200 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.748159 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rljl7" podStartSLOduration=6.917019208 podStartE2EDuration="35.748146849s" totalImagesPullingTime="28.831127641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:57.747933977 +0000 UTC m=+36.845832341" watchObservedRunningTime="2026-09-10 17:13:57.748146849 +0000 UTC m=+36.846045210" Sep 10 17:13:57.951632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:57.951597 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:57.951776 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:57.951745 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:13:57.951835 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:57.951823 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls podName:c81cc54e-dfe7-45a1-a04c-b9abba50450d nodeName:}" failed. No retries permitted until 2026-09-10 17:13:59.951801083 +0000 UTC m=+39.049699422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls") pod "dns-default-lsk5m" (UID: "c81cc54e-dfe7-45a1-a04c-b9abba50450d") : secret "dns-default-metrics-tls" not found Sep 10 17:13:59.731173 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:59.731138 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9549r" event={"ID":"e8bd1e76-8725-4094-a2f5-b4d94d75abf7","Type":"ContainerStarted","Data":"46cb86c321bdfc133788a9f32bb910ad6c6b4ddc543126869f5ef81574cb9ff2"} Sep 10 17:13:59.745472 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:59.745428 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9549r" podStartSLOduration=1.41760497 podStartE2EDuration="3.745413578s" totalImagesPullingTime="2.327808608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:59.74467691 +0000 UTC m=+38.842575282" watchObservedRunningTime="2026-09-10 17:13:59.745413578 +0000 UTC m=+38.843311959" Sep 10 17:13:59.966583 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:13:59.966551 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:13:59.966705 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:59.966650 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:13:59.966705 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:13:59.966695 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls podName:c81cc54e-dfe7-45a1-a04c-b9abba50450d nodeName:}" failed. No retries permitted until 2026-09-10 17:14:03.966682897 +0000 UTC m=+43.064581236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls") pod "dns-default-lsk5m" (UID: "c81cc54e-dfe7-45a1-a04c-b9abba50450d") : secret "dns-default-metrics-tls" not found Sep 10 17:14:03.995083 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:03.995029 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:14:03.995591 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:03.995194 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:14:03.995591 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:03.995273 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls podName:c81cc54e-dfe7-45a1-a04c-b9abba50450d nodeName:}" failed. No retries permitted until 2026-09-10 17:14:11.995254074 +0000 UTC m=+51.093152435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls") pod "dns-default-lsk5m" (UID: "c81cc54e-dfe7-45a1-a04c-b9abba50450d") : secret "dns-default-metrics-tls" not found Sep 10 17:14:12.048369 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:12.048334 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:14:12.048819 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:12.048497 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:14:12.048819 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:12.048562 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls podName:c81cc54e-dfe7-45a1-a04c-b9abba50450d nodeName:}" failed. No retries permitted until 2026-09-10 17:14:28.048546523 +0000 UTC m=+67.146444862 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls") pod "dns-default-lsk5m" (UID: "c81cc54e-dfe7-45a1-a04c-b9abba50450d") : secret "dns-default-metrics-tls" not found Sep 10 17:14:20.706446 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:20.706415 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n64hg" Sep 10 17:14:26.039603 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.039569 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:14:26.039984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.039622 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:14:26.041696 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.041670 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 17:14:26.041819 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.041714 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 17:14:26.050275 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:26.050255 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 17:14:26.050362 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:26.050313 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs podName:cb911006-5fc7-45f2-b42f-b44ccb6d0ac8 nodeName:}" failed. No retries permitted until 2026-09-10 17:15:30.050298222 +0000 UTC m=+129.148196561 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs") pod "network-metrics-daemon-tgzqt" (UID: "cb911006-5fc7-45f2-b42f-b44ccb6d0ac8") : secret "metrics-daemon-secret" not found Sep 10 17:14:26.053260 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.053237 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d2370b0a-f5f3-4ffb-a627-7a6cceae329e-original-pull-secret\") pod \"global-pull-secret-syncer-mrzvh\" (UID: \"d2370b0a-f5f3-4ffb-a627-7a6cceae329e\") " pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:14:26.139991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.139967 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:14:26.142401 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.142387 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 17:14:26.152798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.152781 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 17:14:26.163216 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.163198 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6wb\" (UniqueName: \"kubernetes.io/projected/83f7f6b7-ad11-4b89-a118-b31dc14fcb8e-kube-api-access-gn6wb\") pod \"network-check-target-k87t9\" (UID: \"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e\") " pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:14:26.194391 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.194365 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g7xkh\"" Sep 10 17:14:26.203227 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.203210 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:14:26.205447 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.205430 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mrzvh" Sep 10 17:14:26.509481 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.509448 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k87t9"] Sep 10 17:14:26.512980 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:14:26.512955 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f7f6b7_ad11_4b89_a118_b31dc14fcb8e.slice/crio-88c4be5c1185742e7226e9597087a03fc8ff7fdfc2846fbfdf378b770163c6e7 WatchSource:0}: Error finding container 88c4be5c1185742e7226e9597087a03fc8ff7fdfc2846fbfdf378b770163c6e7: Status 404 returned error can't find the container with id 88c4be5c1185742e7226e9597087a03fc8ff7fdfc2846fbfdf378b770163c6e7 Sep 10 17:14:26.524555 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.524532 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mrzvh"] Sep 10 17:14:26.527239 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:14:26.527217 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2370b0a_f5f3_4ffb_a627_7a6cceae329e.slice/crio-aa7b93118c428b6d6ae903d09d807f051d7362e3d64b39054234e3e80e5c21af WatchSource:0}: Error finding container aa7b93118c428b6d6ae903d09d807f051d7362e3d64b39054234e3e80e5c21af: Status 404 returned error can't find the container with id aa7b93118c428b6d6ae903d09d807f051d7362e3d64b39054234e3e80e5c21af Sep 10 17:14:26.795273 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.795241 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mrzvh" event={"ID":"d2370b0a-f5f3-4ffb-a627-7a6cceae329e","Type":"ContainerStarted","Data":"aa7b93118c428b6d6ae903d09d807f051d7362e3d64b39054234e3e80e5c21af"} Sep 10 17:14:26.796331 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:26.796304 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k87t9" event={"ID":"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e","Type":"ContainerStarted","Data":"88c4be5c1185742e7226e9597087a03fc8ff7fdfc2846fbfdf378b770163c6e7"} Sep 10 17:14:28.050724 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:28.050686 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:14:28.051299 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:28.050860 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 17:14:28.051299 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:28.050937 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls podName:c81cc54e-dfe7-45a1-a04c-b9abba50450d nodeName:}" failed. No retries permitted until 2026-09-10 17:15:00.050916983 +0000 UTC m=+99.148815328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls") pod "dns-default-lsk5m" (UID: "c81cc54e-dfe7-45a1-a04c-b9abba50450d") : secret "dns-default-metrics-tls" not found Sep 10 17:14:31.811180 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:31.811137 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mrzvh" event={"ID":"d2370b0a-f5f3-4ffb-a627-7a6cceae329e","Type":"ContainerStarted","Data":"7e6e4338945466fb4e08a36ab80805346efc4f3df407215c164fc66550670988"} Sep 10 17:14:31.812688 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:31.812663 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k87t9" event={"ID":"83f7f6b7-ad11-4b89-a118-b31dc14fcb8e","Type":"ContainerStarted","Data":"bfb78b37b6a8bb63ced16e41f71313916e0f6aeb662ff2c028e723e895265cd4"} Sep 10 17:14:31.812796 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:31.812785 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:14:31.825156 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:31.825111 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mrzvh" podStartSLOduration=65.431294325 podStartE2EDuration="1m9.825094055s" totalImagesPullingTime="4.39379973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:31.824705946 +0000 UTC m=+70.922604315" watchObservedRunningTime="2026-09-10 17:14:31.825094055 +0000 UTC m=+70.922992417" Sep 10 17:14:31.838400 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:31.838366 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k87t9" podStartSLOduration=65.434034689 podStartE2EDuration="1m9.838352473s" totalImagesPullingTime="4.404317784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:31.838289672 +0000 UTC m=+70.936188056" watchObservedRunningTime="2026-09-10 17:14:31.838352473 +0000 UTC m=+70.936250835" Sep 10 17:14:40.968197 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.968150 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-lv4gt"] Sep 10 17:14:40.971235 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.971217 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l"] Sep 10 17:14:40.971386 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.971364 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-lv4gt" Sep 10 17:14:40.973769 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.973742 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-klvpt\"" Sep 10 17:14:40.973882 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.973778 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:40.975504 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.975484 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 10 17:14:40.975504 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.975501 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 10 17:14:40.975656 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.975622 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 10 17:14:40.975656 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.975633 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:14:40.975656 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.975639 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-q8rvq\"" Sep 10 17:14:40.981824 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.981799 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l"] Sep 10 17:14:40.982460 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:40.982442 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-lv4gt"] Sep 10 17:14:41.038734 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.038713 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vnf\" (UniqueName: \"kubernetes.io/projected/f3eec8c0-6e8a-4498-b83b-23f9d8740fc4-kube-api-access-z8vnf\") pod \"network-check-source-685456655f-lv4gt\" (UID: \"f3eec8c0-6e8a-4498-b83b-23f9d8740fc4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-lv4gt" Sep 10 17:14:41.038830 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.038742 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b96006c6-5198-45c0-ad2f-aa8a703934e6-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bdf5l\" (UID: \"b96006c6-5198-45c0-ad2f-aa8a703934e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.038830 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.038781 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8txr\" (UniqueName: \"kubernetes.io/projected/b96006c6-5198-45c0-ad2f-aa8a703934e6-kube-api-access-v8txr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bdf5l\" (UID: \"b96006c6-5198-45c0-ad2f-aa8a703934e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.038906 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.038830 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b96006c6-5198-45c0-ad2f-aa8a703934e6-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bdf5l\" (UID: \"b96006c6-5198-45c0-ad2f-aa8a703934e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.076503 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.076480 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj"] Sep 10 17:14:41.079342 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.079324 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.082088 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.082058 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 17:14:41.082186 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.082099 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 17:14:41.082788 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.082774 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 17:14:41.082854 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.082834 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 17:14:41.082959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.082946 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bpdnl\"" Sep 10 17:14:41.092078 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.092061 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj"] Sep 10 17:14:41.139339 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.139316 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8txr\" (UniqueName: \"kubernetes.io/projected/b96006c6-5198-45c0-ad2f-aa8a703934e6-kube-api-access-v8txr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bdf5l\" (UID: \"b96006c6-5198-45c0-ad2f-aa8a703934e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.139420 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.139343 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b96006c6-5198-45c0-ad2f-aa8a703934e6-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bdf5l\" (UID: \"b96006c6-5198-45c0-ad2f-aa8a703934e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.139420 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.139378 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.139530 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.139480 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjlqz\" (UniqueName: \"kubernetes.io/projected/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-kube-api-access-gjlqz\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.139530 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.139520 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vnf\" (UniqueName: \"kubernetes.io/projected/f3eec8c0-6e8a-4498-b83b-23f9d8740fc4-kube-api-access-z8vnf\") pod \"network-check-source-685456655f-lv4gt\" (UID: \"f3eec8c0-6e8a-4498-b83b-23f9d8740fc4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-lv4gt" Sep 10 17:14:41.139599 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.139540 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.139599 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.139562 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b96006c6-5198-45c0-ad2f-aa8a703934e6-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bdf5l\" (UID: \"b96006c6-5198-45c0-ad2f-aa8a703934e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.140015 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.139987 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b96006c6-5198-45c0-ad2f-aa8a703934e6-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bdf5l\" (UID: \"b96006c6-5198-45c0-ad2f-aa8a703934e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.141428 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.141409 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b96006c6-5198-45c0-ad2f-aa8a703934e6-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bdf5l\" (UID: \"b96006c6-5198-45c0-ad2f-aa8a703934e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.150933 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.150915 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8txr\" (UniqueName: \"kubernetes.io/projected/b96006c6-5198-45c0-ad2f-aa8a703934e6-kube-api-access-v8txr\") pod \"kube-storage-version-migrator-operator-8d7dd4675-bdf5l\" (UID: \"b96006c6-5198-45c0-ad2f-aa8a703934e6\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.152292 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.152269 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vnf\" (UniqueName: \"kubernetes.io/projected/f3eec8c0-6e8a-4498-b83b-23f9d8740fc4-kube-api-access-z8vnf\") pod \"network-check-source-685456655f-lv4gt\" (UID: \"f3eec8c0-6e8a-4498-b83b-23f9d8740fc4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-lv4gt" Sep 10 17:14:41.167173 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.167152 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-bf89c887b-crb9s"] Sep 10 17:14:41.170092 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.170079 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.172115 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.172094 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 17:14:41.172234 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.172121 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 17:14:41.172234 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.172094 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jcxvb\"" Sep 10 17:14:41.172234 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.172150 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 17:14:41.172234 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.172169 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 17:14:41.172234 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.172173 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 17:14:41.172530 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.172517 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 17:14:41.184201 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.184183 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bf89c887b-crb9s"] Sep 10 17:14:41.240461 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.240441 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-stats-auth\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.240553 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.240483 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.240553 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.240500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjlqz\" (UniqueName: \"kubernetes.io/projected/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-kube-api-access-gjlqz\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.240553 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.240529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.240717 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.240600 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnskw\" (UniqueName: \"kubernetes.io/projected/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-kube-api-access-mnskw\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.240717 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.240635 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.240717 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.240702 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.240874 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.240724 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:41.240874 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.240782 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls podName:0067abc0-3ae9-4e8e-b736-e0a48a9fd797 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:41.740764656 +0000 UTC m=+80.838663002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-jbpqj" (UID: "0067abc0-3ae9-4e8e-b736-e0a48a9fd797") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:41.240874 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.240744 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-default-certificate\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.241256 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.241235 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.256514 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.256489 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjlqz\" (UniqueName: \"kubernetes.io/projected/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-kube-api-access-gjlqz\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.281287 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.281269 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-lv4gt" Sep 10 17:14:41.287024 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.287008 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" Sep 10 17:14:41.341349 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.341301 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-stats-auth\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.341462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.341430 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.341531 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.341463 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnskw\" (UniqueName: \"kubernetes.io/projected/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-kube-api-access-mnskw\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.341590 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.341557 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.341590 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.341586 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-default-certificate\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.342263 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.341962 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:41.841936825 +0000 UTC m=+80.939835184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : configmap references non-existent config key: service-ca.crt Sep 10 17:14:41.342263 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.342054 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:14:41.342263 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.342124 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:41.842106832 +0000 UTC m=+80.940005188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : secret "router-metrics-certs-default" not found Sep 10 17:14:41.345246 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.345200 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-stats-auth\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.345909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.345859 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-default-certificate\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.351822 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.351799 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnskw\" (UniqueName: \"kubernetes.io/projected/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-kube-api-access-mnskw\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.546156 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.546082 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-lv4gt"] Sep 10 17:14:41.550010 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:14:41.549984 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3eec8c0_6e8a_4498_b83b_23f9d8740fc4.slice/crio-46116b3219f195ce4a5511af705cf957c10cdef9c988426194fae96ba882ef52 WatchSource:0}: Error finding container 46116b3219f195ce4a5511af705cf957c10cdef9c988426194fae96ba882ef52: Status 404 returned error can't find the container with id 46116b3219f195ce4a5511af705cf957c10cdef9c988426194fae96ba882ef52 Sep 10 17:14:41.557213 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.557190 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l"] Sep 10 17:14:41.559903 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:14:41.559881 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96006c6_5198_45c0_ad2f_aa8a703934e6.slice/crio-65ef5c84d1dc65e59c720061368114f9cd688508023dd54aa563ab4cd7f328d2 WatchSource:0}: Error finding container 65ef5c84d1dc65e59c720061368114f9cd688508023dd54aa563ab4cd7f328d2: Status 404 returned error can't find the container with id 65ef5c84d1dc65e59c720061368114f9cd688508023dd54aa563ab4cd7f328d2 Sep 10 17:14:41.743651 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.743618 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:41.743790 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.743741 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:41.743851 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.743818 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls podName:0067abc0-3ae9-4e8e-b736-e0a48a9fd797 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:42.743785509 +0000 UTC m=+81.841683863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-jbpqj" (UID: "0067abc0-3ae9-4e8e-b736-e0a48a9fd797") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:41.839072 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.838986 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-lv4gt" event={"ID":"f3eec8c0-6e8a-4498-b83b-23f9d8740fc4","Type":"ContainerStarted","Data":"36be4ceb5dc75f35d0596b881e45f16d755007d0f7fa3d603d5a14f70f2447a7"} Sep 10 17:14:41.839072 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.839047 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-lv4gt" event={"ID":"f3eec8c0-6e8a-4498-b83b-23f9d8740fc4","Type":"ContainerStarted","Data":"46116b3219f195ce4a5511af705cf957c10cdef9c988426194fae96ba882ef52"} Sep 10 17:14:41.842270 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.842240 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" event={"ID":"b96006c6-5198-45c0-ad2f-aa8a703934e6","Type":"ContainerStarted","Data":"65ef5c84d1dc65e59c720061368114f9cd688508023dd54aa563ab4cd7f328d2"} Sep 10 17:14:41.843990 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.843970 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.844112 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.844013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:41.844186 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.844118 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:14:41.844186 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.844122 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:42.844109903 +0000 UTC m=+81.942008241 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : configmap references non-existent config key: service-ca.crt Sep 10 17:14:41.844186 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:41.844177 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:42.844164184 +0000 UTC m=+81.942062522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : secret "router-metrics-certs-default" not found Sep 10 17:14:41.854173 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:41.854137 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-lv4gt" podStartSLOduration=1.854119321 podStartE2EDuration="1.854119321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:41.853486265 +0000 UTC m=+80.951384626" watchObservedRunningTime="2026-09-10 17:14:41.854119321 +0000 UTC m=+80.952017681" Sep 10 17:14:42.750889 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:42.750856 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:42.751326 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:42.750999 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:42.751326 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:42.751080 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls podName:0067abc0-3ae9-4e8e-b736-e0a48a9fd797 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:44.751061996 +0000 UTC m=+83.848960341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-jbpqj" (UID: "0067abc0-3ae9-4e8e-b736-e0a48a9fd797") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:42.852105 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:42.852074 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:42.852270 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:42.852152 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:42.852270 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:42.852258 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:44.852235494 +0000 UTC m=+83.950133834 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : configmap references non-existent config key: service-ca.crt Sep 10 17:14:42.852393 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:42.852274 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:14:42.852393 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:42.852320 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:44.852305969 +0000 UTC m=+83.950204327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : secret "router-metrics-certs-default" not found Sep 10 17:14:43.850401 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:43.850291 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" event={"ID":"b96006c6-5198-45c0-ad2f-aa8a703934e6","Type":"ContainerStarted","Data":"af23994bdaf02aaeba7734c3c4a747df06e8b53426c690bf0b97e39e320f50d4"} Sep 10 17:14:43.865955 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:43.865912 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" podStartSLOduration=2.261255708 podStartE2EDuration="3.865898042s" totalImagesPullingTime="1.604642334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:43.864980776 +0000 UTC m=+82.962879136" watchObservedRunningTime="2026-09-10 17:14:43.865898042 +0000 UTC m=+82.963796403" Sep 10 17:14:44.768326 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.768287 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:44.768519 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:44.768410 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:44.768519 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:44.768467 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls podName:0067abc0-3ae9-4e8e-b736-e0a48a9fd797 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:48.768450568 +0000 UTC m=+87.866348908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-jbpqj" (UID: "0067abc0-3ae9-4e8e-b736-e0a48a9fd797") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:44.841520 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.841486 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48"] Sep 10 17:14:44.844616 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.844598 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48" Sep 10 17:14:44.846601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.846579 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 17:14:44.847214 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.847192 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-n79gb\"" Sep 10 17:14:44.847214 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.847207 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 17:14:44.855101 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.855080 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48"] Sep 10 17:14:44.868671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.868650 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:44.868754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.868699 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:44.868799 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:44.868781 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:14:44.868843 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:44.868832 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:48.868817049 +0000 UTC m=+87.966715389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : configmap references non-existent config key: service-ca.crt Sep 10 17:14:44.868884 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:44.868852 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:48.868844782 +0000 UTC m=+87.966743121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : secret "router-metrics-certs-default" not found Sep 10 17:14:44.969605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:44.969583 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p66dd\" (UniqueName: \"kubernetes.io/projected/15c87953-16b7-4c34-a7b0-eb3b79b09fa2-kube-api-access-p66dd\") pod \"migrator-fdbb8fdff-2tz48\" (UID: \"15c87953-16b7-4c34-a7b0-eb3b79b09fa2\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48" Sep 10 17:14:45.070434 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:45.070387 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p66dd\" (UniqueName: \"kubernetes.io/projected/15c87953-16b7-4c34-a7b0-eb3b79b09fa2-kube-api-access-p66dd\") pod \"migrator-fdbb8fdff-2tz48\" (UID: \"15c87953-16b7-4c34-a7b0-eb3b79b09fa2\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48" Sep 10 17:14:45.078128 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:45.078107 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p66dd\" (UniqueName: \"kubernetes.io/projected/15c87953-16b7-4c34-a7b0-eb3b79b09fa2-kube-api-access-p66dd\") pod \"migrator-fdbb8fdff-2tz48\" (UID: \"15c87953-16b7-4c34-a7b0-eb3b79b09fa2\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48" Sep 10 17:14:45.154337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:45.154314 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48" Sep 10 17:14:45.422012 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:45.421944 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48"] Sep 10 17:14:45.424883 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:14:45.424857 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c87953_16b7_4c34_a7b0_eb3b79b09fa2.slice/crio-d3c7cd5c3da2b04cf21d19d9acdbd5dfdb1b3b830c6591d94d49fd234f69872e WatchSource:0}: Error finding container d3c7cd5c3da2b04cf21d19d9acdbd5dfdb1b3b830c6591d94d49fd234f69872e: Status 404 returned error can't find the container with id d3c7cd5c3da2b04cf21d19d9acdbd5dfdb1b3b830c6591d94d49fd234f69872e Sep 10 17:14:45.858737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:45.858701 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48" event={"ID":"15c87953-16b7-4c34-a7b0-eb3b79b09fa2","Type":"ContainerStarted","Data":"d3c7cd5c3da2b04cf21d19d9acdbd5dfdb1b3b830c6591d94d49fd234f69872e"} Sep 10 17:14:46.008650 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.008619 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6qz6j"] Sep 10 17:14:46.011794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.011773 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.013946 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.013923 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 17:14:46.014082 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.013947 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2hvq7\"" Sep 10 17:14:46.014082 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.013970 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 17:14:46.014378 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.014356 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 17:14:46.014378 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.014364 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 17:14:46.031116 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.031093 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qz6j"] Sep 10 17:14:46.078704 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.078674 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d07b5b9b-b239-4200-8984-753fe31b0551-crio-socket\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.078846 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.078714 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d07b5b9b-b239-4200-8984-753fe31b0551-data-volume\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.078846 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.078742 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.078846 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.078807 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.078964 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.078845 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vdz\" (UniqueName: \"kubernetes.io/projected/d07b5b9b-b239-4200-8984-753fe31b0551-kube-api-access-t8vdz\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.180090 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.180001 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d07b5b9b-b239-4200-8984-753fe31b0551-crio-socket\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.180090 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.180056 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d07b5b9b-b239-4200-8984-753fe31b0551-data-volume\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.180291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.180082 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.180291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.180151 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d07b5b9b-b239-4200-8984-753fe31b0551-crio-socket\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.180291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.180159 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.180291 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:46.180209 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:14:46.180291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.180234 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vdz\" (UniqueName: \"kubernetes.io/projected/d07b5b9b-b239-4200-8984-753fe31b0551-kube-api-access-t8vdz\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.180291 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:46.180258 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:46.680243963 +0000 UTC m=+85.778142302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : configmap "kube-rbac-proxy" not found Sep 10 17:14:46.180576 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:46.180293 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:14:46.180576 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:46.180366 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:46.680346334 +0000 UTC m=+85.778244690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : secret "insights-runtime-extractor-tls" not found Sep 10 17:14:46.180576 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.180416 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d07b5b9b-b239-4200-8984-753fe31b0551-data-volume\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.195824 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.195792 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vdz\" (UniqueName: \"kubernetes.io/projected/d07b5b9b-b239-4200-8984-753fe31b0551-kube-api-access-t8vdz\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.682943 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.682905 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.683109 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.682965 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:46.683109 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:46.683055 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:14:46.683109 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:46.683075 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:14:46.683213 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:46.683112 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:47.683098509 +0000 UTC m=+86.780996862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : configmap "kube-rbac-proxy" not found Sep 10 17:14:46.683213 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:46.683131 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:47.683117857 +0000 UTC m=+86.781016214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : secret "insights-runtime-extractor-tls" not found Sep 10 17:14:46.864416 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.864381 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48" event={"ID":"15c87953-16b7-4c34-a7b0-eb3b79b09fa2","Type":"ContainerStarted","Data":"1022e7a789f68e6e68a6b4da33e65175cebb779bd14f71cf75b239a87e69a715"} Sep 10 17:14:46.864840 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.864431 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48" event={"ID":"15c87953-16b7-4c34-a7b0-eb3b79b09fa2","Type":"ContainerStarted","Data":"b3b0c380511c34bf6af58dc2f37089da0e49eff6c2b90843f0812009ed28299f"} Sep 10 17:14:46.881835 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:46.881786 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-2tz48" podStartSLOduration=2.005463373 podStartE2EDuration="2.881771822s" totalImagesPullingTime="876.308449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:46.880954835 +0000 UTC m=+85.978853197" watchObservedRunningTime="2026-09-10 17:14:46.881771822 +0000 UTC m=+85.979670182" Sep 10 17:14:47.689491 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:47.689452 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:47.689685 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:47.689531 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:47.689685 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:47.689595 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:14:47.689685 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:47.689615 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:14:47.689685 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:47.689665 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:49.689653476 +0000 UTC m=+88.787551815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : secret "insights-runtime-extractor-tls" not found Sep 10 17:14:47.689685 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:47.689680 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:49.689673733 +0000 UTC m=+88.787572071 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : configmap "kube-rbac-proxy" not found Sep 10 17:14:48.798637 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:48.798601 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:48.799068 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:48.798783 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:48.799068 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:48.798866 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls podName:0067abc0-3ae9-4e8e-b736-e0a48a9fd797 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:56.798842761 +0000 UTC m=+95.896741120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-jbpqj" (UID: "0067abc0-3ae9-4e8e-b736-e0a48a9fd797") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:48.899503 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:48.899474 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:48.899633 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:48.899569 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:48.899701 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:48.899625 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:56.899609484 +0000 UTC m=+95.997507850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : configmap references non-existent config key: service-ca.crt Sep 10 17:14:48.899701 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:48.899686 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:14:48.899799 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:48.899742 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:56.8997254 +0000 UTC m=+95.997623756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : secret "router-metrics-certs-default" not found Sep 10 17:14:48.935581 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:48.935560 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x275d_6e5ea32f-27b0-4267-98b1-9ceb809f9f44/dns-node-resolver/0.log" Sep 10 17:14:49.705577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:49.705546 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:49.705758 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:49.705599 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:49.705758 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:49.705657 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:14:49.705758 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:49.705673 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:14:49.705758 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:49.705704 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:53.705692963 +0000 UTC m=+92.803591315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : configmap "kube-rbac-proxy" not found Sep 10 17:14:49.705758 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:49.705730 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:14:53.705710488 +0000 UTC m=+92.803608827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : secret "insights-runtime-extractor-tls" not found Sep 10 17:14:49.735592 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:49.735565 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zdrnp_a38b50d7-4bd2-4605-8896-fd0d0f5b35d8/node-ca/0.log" Sep 10 17:14:50.538118 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:50.538091 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9549r_e8bd1e76-8725-4094-a2f5-b4d94d75abf7/serve-healthcheck-canary/0.log" Sep 10 17:14:50.935290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:50.935217 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-2tz48_15c87953-16b7-4c34-a7b0-eb3b79b09fa2/migrator/0.log" Sep 10 17:14:51.134604 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:51.134579 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-2tz48_15c87953-16b7-4c34-a7b0-eb3b79b09fa2/graceful-termination/0.log" Sep 10 17:14:51.337066 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:51.337043 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-bdf5l_b96006c6-5198-45c0-ad2f-aa8a703934e6/kube-storage-version-migrator-operator/0.log" Sep 10 17:14:53.731458 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:53.731420 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:53.731880 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:53.731490 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:14:53.731880 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:53.731571 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:14:53.731880 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:53.731630 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:15:01.731614492 +0000 UTC m=+100.829512834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : secret "insights-runtime-extractor-tls" not found Sep 10 17:14:53.731880 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:53.731571 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 17:14:53.731880 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:53.731703 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:15:01.731690538 +0000 UTC m=+100.829588891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : configmap "kube-rbac-proxy" not found Sep 10 17:14:56.855672 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:56.855631 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:14:56.856082 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:56.855747 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:56.856082 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:56.855798 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls podName:0067abc0-3ae9-4e8e-b736-e0a48a9fd797 nodeName:}" failed. No retries permitted until 2026-09-10 17:15:12.855784775 +0000 UTC m=+111.953683114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-jbpqj" (UID: "0067abc0-3ae9-4e8e-b736-e0a48a9fd797") : secret "cluster-monitoring-operator-tls" not found Sep 10 17:14:56.956378 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:56.956346 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:56.956531 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:14:56.956415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:14:56.956531 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:56.956485 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 17:14:56.956614 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:56.956531 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:15:12.956512608 +0000 UTC m=+112.054410953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : configmap references non-existent config key: service-ca.crt Sep 10 17:14:56.956614 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:14:56.956545 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs podName:52ef40d5-e77b-4b0d-acdd-cebf3147dcb4 nodeName:}" failed. No retries permitted until 2026-09-10 17:15:12.956539041 +0000 UTC m=+112.054437380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs") pod "router-default-bf89c887b-crb9s" (UID: "52ef40d5-e77b-4b0d-acdd-cebf3147dcb4") : secret "router-metrics-certs-default" not found Sep 10 17:15:00.078690 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:00.078649 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:15:00.080859 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:00.080831 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c81cc54e-dfe7-45a1-a04c-b9abba50450d-metrics-tls\") pod \"dns-default-lsk5m\" (UID: \"c81cc54e-dfe7-45a1-a04c-b9abba50450d\") " pod="openshift-dns/dns-default-lsk5m" Sep 10 17:15:00.106801 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:00.106777 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2rm4x\"" Sep 10 17:15:00.115333 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:00.115317 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lsk5m" Sep 10 17:15:00.417957 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:00.417889 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lsk5m"] Sep 10 17:15:00.420957 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:15:00.420931 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81cc54e_dfe7_45a1_a04c_b9abba50450d.slice/crio-3fbb10f735e3ccc960bc64b22bdf3e8ca60fe316735508661e97e2e7f2ca79b4 WatchSource:0}: Error finding container 3fbb10f735e3ccc960bc64b22bdf3e8ca60fe316735508661e97e2e7f2ca79b4: Status 404 returned error can't find the container with id 3fbb10f735e3ccc960bc64b22bdf3e8ca60fe316735508661e97e2e7f2ca79b4 Sep 10 17:15:00.906600 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:00.906561 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lsk5m" event={"ID":"c81cc54e-dfe7-45a1-a04c-b9abba50450d","Type":"ContainerStarted","Data":"3fbb10f735e3ccc960bc64b22bdf3e8ca60fe316735508661e97e2e7f2ca79b4"} Sep 10 17:15:01.793205 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:01.793167 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:15:01.793677 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:01.793239 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:15:01.793677 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:15:01.793320 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 17:15:01.793677 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:15:01.793399 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls podName:d07b5b9b-b239-4200-8984-753fe31b0551 nodeName:}" failed. No retries permitted until 2026-09-10 17:15:17.793377433 +0000 UTC m=+116.891275791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6qz6j" (UID: "d07b5b9b-b239-4200-8984-753fe31b0551") : secret "insights-runtime-extractor-tls" not found Sep 10 17:15:01.793878 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:01.793856 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d07b5b9b-b239-4200-8984-753fe31b0551-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:15:02.817963 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:02.817934 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k87t9" Sep 10 17:15:02.914293 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:02.914268 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lsk5m" event={"ID":"c81cc54e-dfe7-45a1-a04c-b9abba50450d","Type":"ContainerStarted","Data":"08ee0e2b907dced2b9d58953bd6756de2e9b29dd5660da38a86c7b4c9553d40e"} Sep 10 17:15:02.914421 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:02.914319 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lsk5m" event={"ID":"c81cc54e-dfe7-45a1-a04c-b9abba50450d","Type":"ContainerStarted","Data":"97b4fdfa4cdd7226de164f3ec3feecffcf33e21399ec4221d23cdb1f44ab4068"} Sep 10 17:15:02.914472 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:02.914441 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lsk5m" Sep 10 17:15:02.930797 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:02.930758 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lsk5m" podStartSLOduration=64.938038709 podStartE2EDuration="1m6.930743849s" totalImagesPullingTime="1.99270514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:02.929580541 +0000 UTC m=+102.027478914" watchObservedRunningTime="2026-09-10 17:15:02.930743849 +0000 UTC m=+102.028642210" Sep 10 17:15:08.433510 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.433449 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5"] Sep 10 17:15:08.435634 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.435615 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.438783 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.438748 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 10 17:15:08.439071 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.439048 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 17:15:08.439158 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.439140 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 17:15:08.439340 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.439322 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 10 17:15:08.439409 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.439346 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 17:15:08.440104 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.440087 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 10 17:15:08.440197 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.440181 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 10 17:15:08.451961 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.451927 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5"] Sep 10 17:15:08.548401 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.548357 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-ca\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.548613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.548415 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-hub\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.548613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.548533 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ccd8c41-6285-447d-97bb-1d8ee301c96e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.548613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.548579 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.548805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.548787 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnttm\" (UniqueName: \"kubernetes.io/projected/1ccd8c41-6285-447d-97bb-1d8ee301c96e-kube-api-access-bnttm\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.548878 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.548829 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.649885 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.649845 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ccd8c41-6285-447d-97bb-1d8ee301c96e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.649885 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.649901 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.650183 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.649960 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnttm\" (UniqueName: \"kubernetes.io/projected/1ccd8c41-6285-447d-97bb-1d8ee301c96e-kube-api-access-bnttm\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.650183 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.649993 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.650183 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.650082 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-ca\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.650183 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.650122 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-hub\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.650731 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.650702 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ccd8c41-6285-447d-97bb-1d8ee301c96e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.652807 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.652778 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-ca\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.653000 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.652805 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.653095 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.653016 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.653334 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.653312 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1ccd8c41-6285-447d-97bb-1d8ee301c96e-hub\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.658967 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.658946 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnttm\" (UniqueName: \"kubernetes.io/projected/1ccd8c41-6285-447d-97bb-1d8ee301c96e-kube-api-access-bnttm\") pod \"cluster-proxy-proxy-agent-5c6d68b4c5-4brm5\" (UID: \"1ccd8c41-6285-447d-97bb-1d8ee301c96e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:08.753652 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:08.753613 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" Sep 10 17:15:09.028188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:09.028098 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5"] Sep 10 17:15:09.031391 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:15:09.031355 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ccd8c41_6285_447d_97bb_1d8ee301c96e.slice/crio-f412b4d0b8b1ef863ac4a582995b0bf1b8641fa57ec7436c40e8ae8ff67febc9 WatchSource:0}: Error finding container f412b4d0b8b1ef863ac4a582995b0bf1b8641fa57ec7436c40e8ae8ff67febc9: Status 404 returned error can't find the container with id f412b4d0b8b1ef863ac4a582995b0bf1b8641fa57ec7436c40e8ae8ff67febc9 Sep 10 17:15:09.938253 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:09.938200 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" event={"ID":"1ccd8c41-6285-447d-97bb-1d8ee301c96e","Type":"ContainerStarted","Data":"f412b4d0b8b1ef863ac4a582995b0bf1b8641fa57ec7436c40e8ae8ff67febc9"} Sep 10 17:15:12.886108 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.885977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:15:12.888424 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.888397 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0067abc0-3ae9-4e8e-b736-e0a48a9fd797-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-jbpqj\" (UID: \"0067abc0-3ae9-4e8e-b736-e0a48a9fd797\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:15:12.920405 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.920353 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lsk5m" Sep 10 17:15:12.952150 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.952109 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" event={"ID":"1ccd8c41-6285-447d-97bb-1d8ee301c96e","Type":"ContainerStarted","Data":"85eecebdc0ac396b018e40b16b0ffbc90152cbc05b10819eedc7a89d9445042a"} Sep 10 17:15:12.952441 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.952169 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" event={"ID":"1ccd8c41-6285-447d-97bb-1d8ee301c96e","Type":"ContainerStarted","Data":"8115c144ee2903e4da642d39e37228cae64817bd8c50a2a5143c67e368e70211"} Sep 10 17:15:12.952441 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.952197 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" event={"ID":"1ccd8c41-6285-447d-97bb-1d8ee301c96e","Type":"ContainerStarted","Data":"499d9d9a85854bf054a638ae683b41461400c5d4d781c46cc195e2e2a46db53a"} Sep 10 17:15:12.972575 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.972488 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c6d68b4c5-4brm5" podStartSLOduration=1.47731071 podStartE2EDuration="4.972469198s" totalImagesPullingTime="3.495158488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:12.972165896 +0000 UTC m=+112.070064257" watchObservedRunningTime="2026-09-10 17:15:12.972469198 +0000 UTC m=+112.070367558" Sep 10 17:15:12.987007 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.986970 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:15:12.987206 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.987023 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:15:12.987661 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.987641 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-service-ca-bundle\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:15:12.989502 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:12.989471 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ef40d5-e77b-4b0d-acdd-cebf3147dcb4-metrics-certs\") pod \"router-default-bf89c887b-crb9s\" (UID: \"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4\") " pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:15:13.187325 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:13.187226 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" Sep 10 17:15:13.278628 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:13.278589 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:15:13.458175 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:13.458092 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj"] Sep 10 17:15:13.460906 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:15:13.460875 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0067abc0_3ae9_4e8e_b736_e0a48a9fd797.slice/crio-4657a833530dc1193149c5ba3ad3a764045a6a3a105064b7565373f975690450 WatchSource:0}: Error finding container 4657a833530dc1193149c5ba3ad3a764045a6a3a105064b7565373f975690450: Status 404 returned error can't find the container with id 4657a833530dc1193149c5ba3ad3a764045a6a3a105064b7565373f975690450 Sep 10 17:15:13.559101 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:13.555896 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-bf89c887b-crb9s"] Sep 10 17:15:13.562563 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:15:13.562510 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ef40d5_e77b_4b0d_acdd_cebf3147dcb4.slice/crio-d8837ce7e6d166794dc292a184e8fc81f63252a47da23da4ca8062f95bdf8199 WatchSource:0}: Error finding container d8837ce7e6d166794dc292a184e8fc81f63252a47da23da4ca8062f95bdf8199: Status 404 returned error can't find the container with id d8837ce7e6d166794dc292a184e8fc81f63252a47da23da4ca8062f95bdf8199 Sep 10 17:15:13.956200 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:13.956158 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" event={"ID":"0067abc0-3ae9-4e8e-b736-e0a48a9fd797","Type":"ContainerStarted","Data":"4657a833530dc1193149c5ba3ad3a764045a6a3a105064b7565373f975690450"} Sep 10 17:15:13.957590 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:13.957555 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bf89c887b-crb9s" event={"ID":"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4","Type":"ContainerStarted","Data":"7178bf4e82766e41edc7d0a1b516668986adc754654b46a781b6fbc8fa77b8f4"} Sep 10 17:15:13.957715 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:13.957606 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-bf89c887b-crb9s" event={"ID":"52ef40d5-e77b-4b0d-acdd-cebf3147dcb4","Type":"ContainerStarted","Data":"d8837ce7e6d166794dc292a184e8fc81f63252a47da23da4ca8062f95bdf8199"} Sep 10 17:15:13.978503 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:13.978448 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-bf89c887b-crb9s" podStartSLOduration=32.978431376 podStartE2EDuration="32.978431376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:13.977298528 +0000 UTC m=+113.075196913" watchObservedRunningTime="2026-09-10 17:15:13.978431376 +0000 UTC m=+113.076329737" Sep 10 17:15:14.279617 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:14.279575 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:15:14.282899 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:14.282865 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:15:14.961693 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:14.961653 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:15:14.963286 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:14.963264 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-bf89c887b-crb9s" Sep 10 17:15:15.967934 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:15.967892 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" event={"ID":"0067abc0-3ae9-4e8e-b736-e0a48a9fd797","Type":"ContainerStarted","Data":"f7319129f22dc5207f3044e0432f4b5dce40c8097ceb54d892029de324be896e"} Sep 10 17:15:15.993160 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:15.993091 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-jbpqj" podStartSLOduration=33.252595419 podStartE2EDuration="34.993067231s" totalImagesPullingTime="1.740471812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:15.992540984 +0000 UTC m=+115.090439344" watchObservedRunningTime="2026-09-10 17:15:15.993067231 +0000 UTC m=+115.090965594" Sep 10 17:15:17.831124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:17.831069 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:15:17.833492 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:17.833461 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d07b5b9b-b239-4200-8984-753fe31b0551-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qz6j\" (UID: \"d07b5b9b-b239-4200-8984-753fe31b0551\") " pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:15:18.121693 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:18.121596 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qz6j" Sep 10 17:15:18.407678 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:18.407650 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qz6j"] Sep 10 17:15:18.409788 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:15:18.409762 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07b5b9b_b239_4200_8984_753fe31b0551.slice/crio-5abd7f0626620cf26efbbc9abecb0137eca8c0ab560ed03b66dfde7ce578efe9 WatchSource:0}: Error finding container 5abd7f0626620cf26efbbc9abecb0137eca8c0ab560ed03b66dfde7ce578efe9: Status 404 returned error can't find the container with id 5abd7f0626620cf26efbbc9abecb0137eca8c0ab560ed03b66dfde7ce578efe9 Sep 10 17:15:18.980053 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:18.979988 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qz6j" event={"ID":"d07b5b9b-b239-4200-8984-753fe31b0551","Type":"ContainerStarted","Data":"040b30d212c47f8844221c4f23fc90b83b50d75d9f63764f174754a4be4103ac"} Sep 10 17:15:18.980053 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:18.980089 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qz6j" event={"ID":"d07b5b9b-b239-4200-8984-753fe31b0551","Type":"ContainerStarted","Data":"5abd7f0626620cf26efbbc9abecb0137eca8c0ab560ed03b66dfde7ce578efe9"} Sep 10 17:15:19.986192 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:19.986155 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qz6j" event={"ID":"d07b5b9b-b239-4200-8984-753fe31b0551","Type":"ContainerStarted","Data":"0da5a26986abdc30cc02f9c0618e5373ba6fea97fc712446a82e69ae8373e168"} Sep 10 17:15:20.992189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:20.992149 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qz6j" event={"ID":"d07b5b9b-b239-4200-8984-753fe31b0551","Type":"ContainerStarted","Data":"6977802d556394e363b7107aae907705de09ba29aaf2036962b30541aecb2966"} Sep 10 17:15:24.689056 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.688935 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6qz6j" podStartSLOduration=37.700656933 podStartE2EDuration="39.688893802s" totalImagesPullingTime="1.988236869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:14:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:21.033880948 +0000 UTC m=+120.131779308" watchObservedRunningTime="2026-09-10 17:15:24.688893802 +0000 UTC m=+123.786792163" Sep 10 17:15:24.690007 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.689970 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s8qpv"] Sep 10 17:15:24.694168 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.694119 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.697805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.697724 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 17:15:24.698169 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.697768 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 17:15:24.698169 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.697993 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 17:15:24.698169 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.697993 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 17:15:24.698567 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.698544 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gkjjw\"" Sep 10 17:15:24.783797 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.783750 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-sys\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.783797 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.783804 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-wtmp\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.784108 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.783897 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-root\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.784108 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.783927 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-metrics-client-ca\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.784108 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.783954 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-tls\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.784108 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.784012 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.784108 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.784070 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.784108 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.784101 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-textfile\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.784304 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.784123 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spp44\" (UniqueName: \"kubernetes.io/projected/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-kube-api-access-spp44\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.884730 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.884686 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-wtmp\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.884922 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.884859 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-root\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.884922 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.884891 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-wtmp\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.884922 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.884903 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-metrics-client-ca\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885074 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.884945 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-tls\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885074 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.884974 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885074 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.885001 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885074 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.885021 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-root\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885074 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.885065 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-textfile\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.885096 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spp44\" (UniqueName: \"kubernetes.io/projected/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-kube-api-access-spp44\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.885129 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-sys\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.885217 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-sys\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885520 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.885490 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-textfile\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885637 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.885620 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-metrics-client-ca\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.885690 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.885613 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.887596 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.887563 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-tls\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.887721 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.887604 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:24.899572 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:24.899538 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spp44\" (UniqueName: \"kubernetes.io/projected/61f7efb5-13ae-4b0a-ad41-043d0c440ce4-kube-api-access-spp44\") pod \"node-exporter-s8qpv\" (UID: \"61f7efb5-13ae-4b0a-ad41-043d0c440ce4\") " pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:25.010962 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:25.010928 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s8qpv" Sep 10 17:15:25.022175 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:15:25.022135 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f7efb5_13ae_4b0a_ad41_043d0c440ce4.slice/crio-aa78ccaafd90a8d245d2edad0155cd2bdfdc7c9e4c7543ef85c62fea77607808 WatchSource:0}: Error finding container aa78ccaafd90a8d245d2edad0155cd2bdfdc7c9e4c7543ef85c62fea77607808: Status 404 returned error can't find the container with id aa78ccaafd90a8d245d2edad0155cd2bdfdc7c9e4c7543ef85c62fea77607808 Sep 10 17:15:26.012126 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:26.012076 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8qpv" event={"ID":"61f7efb5-13ae-4b0a-ad41-043d0c440ce4","Type":"ContainerStarted","Data":"49798206814cefcbf1bb1a1d9a147b75309ec944e3f3d81f390b7840dff70d4e"} Sep 10 17:15:26.012559 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:26.012152 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8qpv" event={"ID":"61f7efb5-13ae-4b0a-ad41-043d0c440ce4","Type":"ContainerStarted","Data":"aa78ccaafd90a8d245d2edad0155cd2bdfdc7c9e4c7543ef85c62fea77607808"} Sep 10 17:15:27.017322 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:27.017289 2557 generic.go:356] "Generic (PLEG): container finished" podID="61f7efb5-13ae-4b0a-ad41-043d0c440ce4" containerID="49798206814cefcbf1bb1a1d9a147b75309ec944e3f3d81f390b7840dff70d4e" exitCode=0 Sep 10 17:15:27.017704 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:27.017355 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8qpv" event={"ID":"61f7efb5-13ae-4b0a-ad41-043d0c440ce4","Type":"ContainerDied","Data":"49798206814cefcbf1bb1a1d9a147b75309ec944e3f3d81f390b7840dff70d4e"} Sep 10 17:15:28.023569 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:28.023533 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8qpv" event={"ID":"61f7efb5-13ae-4b0a-ad41-043d0c440ce4","Type":"ContainerStarted","Data":"728038d4912d42c8a3846df8c40702b5fd646a466531760f6509e6749b48936c"} Sep 10 17:15:28.024048 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:28.023595 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8qpv" event={"ID":"61f7efb5-13ae-4b0a-ad41-043d0c440ce4","Type":"ContainerStarted","Data":"ad44130f4e430330b8e5cb3dae2e3911e26ad0e625d462bd6b1a8dcbae711f48"} Sep 10 17:15:28.077942 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:28.077881 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s8qpv" podStartSLOduration=3.136841447 podStartE2EDuration="4.077862933s" totalImagesPullingTime="873.463161ms" totalInitContainerRuntime="67.558325ms" isStatefulPod=true podCreationTimestamp="2026-09-10 17:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:28.076849548 +0000 UTC m=+127.174747909" watchObservedRunningTime="2026-09-10 17:15:28.077862933 +0000 UTC m=+127.175761293" Sep 10 17:15:30.129339 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:30.129294 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:15:30.131841 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:30.131810 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb911006-5fc7-45f2-b42f-b44ccb6d0ac8-metrics-certs\") pod \"network-metrics-daemon-tgzqt\" (UID: \"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8\") " pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:15:30.401489 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:30.401387 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n4rqr\"" Sep 10 17:15:30.410090 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:30.410061 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzqt" Sep 10 17:15:30.682608 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:30.682512 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tgzqt"] Sep 10 17:15:30.686296 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:15:30.686268 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb911006_5fc7_45f2_b42f_b44ccb6d0ac8.slice/crio-e83eec0df591ae031184503cc53eb6aec4430789712459b59c25c40b1ef3ed50 WatchSource:0}: Error finding container e83eec0df591ae031184503cc53eb6aec4430789712459b59c25c40b1ef3ed50: Status 404 returned error can't find the container with id e83eec0df591ae031184503cc53eb6aec4430789712459b59c25c40b1ef3ed50 Sep 10 17:15:31.036562 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:31.036521 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgzqt" event={"ID":"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8","Type":"ContainerStarted","Data":"e83eec0df591ae031184503cc53eb6aec4430789712459b59c25c40b1ef3ed50"} Sep 10 17:15:32.041580 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:32.041541 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgzqt" event={"ID":"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8","Type":"ContainerStarted","Data":"afeb5f69047bc981fa370cbf1d67549aa8beca92c507de17d58aa80882e3be66"} Sep 10 17:15:32.042127 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:32.041647 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgzqt" event={"ID":"cb911006-5fc7-45f2-b42f-b44ccb6d0ac8","Type":"ContainerStarted","Data":"8963d20edac2e0f4e5e9bf9606db10faf19e78ed9cd6a69f6437b23bb4b1d6d0"} Sep 10 17:15:32.081318 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:32.081262 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tgzqt" podStartSLOduration=129.211456903 podStartE2EDuration="2m10.081244627s" totalImagesPullingTime="869.787724ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:15:32.0800891 +0000 UTC m=+131.177987460" watchObservedRunningTime="2026-09-10 17:15:32.081244627 +0000 UTC m=+131.179142988" Sep 10 17:15:57.102021 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:57.101985 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bf89c887b-crb9s_52ef40d5-e77b-4b0d-acdd-cebf3147dcb4/router/0.log" Sep 10 17:15:57.119910 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:15:57.119877 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9549r_e8bd1e76-8725-4094-a2f5-b4d94d75abf7/serve-healthcheck-canary/0.log" Sep 10 17:16:09.176084 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:16:09.175962 2557 generic.go:356] "Generic (PLEG): container finished" podID="b96006c6-5198-45c0-ad2f-aa8a703934e6" containerID="af23994bdaf02aaeba7734c3c4a747df06e8b53426c690bf0b97e39e320f50d4" exitCode=0 Sep 10 17:16:09.177569 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:16:09.176677 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" event={"ID":"b96006c6-5198-45c0-ad2f-aa8a703934e6","Type":"ContainerDied","Data":"af23994bdaf02aaeba7734c3c4a747df06e8b53426c690bf0b97e39e320f50d4"} Sep 10 17:16:09.177569 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:16:09.177152 2557 scope.go:117] "RemoveContainer" containerID="af23994bdaf02aaeba7734c3c4a747df06e8b53426c690bf0b97e39e320f50d4" Sep 10 17:16:10.182886 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:16:10.182843 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-bdf5l" event={"ID":"b96006c6-5198-45c0-ad2f-aa8a703934e6","Type":"ContainerStarted","Data":"990db14ea22b287ec8a1d3faea8a110fc56e5c504dd69812ceebef63b411fa7d"} Sep 10 17:18:21.271607 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:18:21.271581 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:18:21.274733 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:18:21.274707 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:18:21.278975 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:18:21.278955 2557 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 17:19:49.709482 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.709404 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428"] Sep 10 17:19:49.711480 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.711458 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" Sep 10 17:19:49.714842 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.714820 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 17:19:49.715555 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.715537 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9lss8\"" Sep 10 17:19:49.715613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.715591 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 17:19:49.724999 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.724976 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428"] Sep 10 17:19:49.897814 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.897773 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d48625f-bf7e-45ba-a9cd-8e42697bc0ab-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q8428\" (UID: \"5d48625f-bf7e-45ba-a9cd-8e42697bc0ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" Sep 10 17:19:49.897993 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.897861 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvdm5\" (UniqueName: \"kubernetes.io/projected/5d48625f-bf7e-45ba-a9cd-8e42697bc0ab-kube-api-access-wvdm5\") pod \"openshift-lws-operator-fd8ccff4c-q8428\" (UID: \"5d48625f-bf7e-45ba-a9cd-8e42697bc0ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" Sep 10 17:19:49.998811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.998778 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvdm5\" (UniqueName: \"kubernetes.io/projected/5d48625f-bf7e-45ba-a9cd-8e42697bc0ab-kube-api-access-wvdm5\") pod \"openshift-lws-operator-fd8ccff4c-q8428\" (UID: \"5d48625f-bf7e-45ba-a9cd-8e42697bc0ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" Sep 10 17:19:49.998993 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.998819 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d48625f-bf7e-45ba-a9cd-8e42697bc0ab-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q8428\" (UID: \"5d48625f-bf7e-45ba-a9cd-8e42697bc0ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" Sep 10 17:19:49.999189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:49.999170 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d48625f-bf7e-45ba-a9cd-8e42697bc0ab-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q8428\" (UID: \"5d48625f-bf7e-45ba-a9cd-8e42697bc0ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" Sep 10 17:19:50.007578 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:50.007555 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvdm5\" (UniqueName: \"kubernetes.io/projected/5d48625f-bf7e-45ba-a9cd-8e42697bc0ab-kube-api-access-wvdm5\") pod \"openshift-lws-operator-fd8ccff4c-q8428\" (UID: \"5d48625f-bf7e-45ba-a9cd-8e42697bc0ab\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" Sep 10 17:19:50.020520 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:50.020491 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" Sep 10 17:19:50.294826 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:50.294747 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428"] Sep 10 17:19:50.298157 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:19:50.298127 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d48625f_bf7e_45ba_a9cd_8e42697bc0ab.slice/crio-7bb42b5ed1de2a42e5141ebee526529a9cf1e5218ee343bc8983649c2c6c2f99 WatchSource:0}: Error finding container 7bb42b5ed1de2a42e5141ebee526529a9cf1e5218ee343bc8983649c2c6c2f99: Status 404 returned error can't find the container with id 7bb42b5ed1de2a42e5141ebee526529a9cf1e5218ee343bc8983649c2c6c2f99 Sep 10 17:19:50.300107 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:50.300089 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:19:50.939058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:50.939008 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" event={"ID":"5d48625f-bf7e-45ba-a9cd-8e42697bc0ab","Type":"ContainerStarted","Data":"7bb42b5ed1de2a42e5141ebee526529a9cf1e5218ee343bc8983649c2c6c2f99"} Sep 10 17:19:52.948594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:52.948548 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" event={"ID":"5d48625f-bf7e-45ba-a9cd-8e42697bc0ab","Type":"ContainerStarted","Data":"a037127d1336f1603e3aed041a51760c93cdf95aa8dc713959fccaf54f0c1809"} Sep 10 17:19:52.972578 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:19:52.972525 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q8428" podStartSLOduration=1.42631384 podStartE2EDuration="3.972509745s" totalImagesPullingTime="2.546195905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:19:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:19:52.970659159 +0000 UTC m=+392.068557522" watchObservedRunningTime="2026-09-10 17:19:52.972509745 +0000 UTC m=+392.070408106" Sep 10 17:20:13.021670 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.021638 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn"] Sep 10 17:20:13.023707 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.023689 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.033183 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.033153 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 17:20:13.033183 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.033165 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 17:20:13.033338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.033155 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 17:20:13.033338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.033217 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 17:20:13.033338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.033242 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xgkxt\"" Sep 10 17:20:13.059551 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.059529 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-manifests\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.059637 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.059555 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.059637 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.059572 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.059637 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.059609 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xth6f\" (UniqueName: \"kubernetes.io/projected/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-kube-api-access-xth6f\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.059637 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.059633 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-tmp\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.081919 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.081891 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn"] Sep 10 17:20:13.160343 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.160319 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-manifests\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.160445 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.160351 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.160445 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.160377 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.160445 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.160435 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xth6f\" (UniqueName: \"kubernetes.io/projected/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-kube-api-access-xth6f\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.160612 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.160465 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-tmp\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.160816 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.160795 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-manifests\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.162612 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.162593 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-tmp\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.162908 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.162892 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.162959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.162943 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.178784 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.178762 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xth6f\" (UniqueName: \"kubernetes.io/projected/ccc1e89d-594c-41fd-a333-8a1ec6c061f8-kube-api-access-xth6f\") pod \"opendatahub-operator-controller-manager-8dc495f6d-twbtn\" (UID: \"ccc1e89d-594c-41fd-a333-8a1ec6c061f8\") " pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.333435 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.333286 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:13.638392 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:13.638317 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn"] Sep 10 17:20:13.643198 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:20:13.643171 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc1e89d_594c_41fd_a333_8a1ec6c061f8.slice/crio-f4985380d83311a4fe0ef175ea31de903dc68c6a4121250a4cc370dbb1ae3daa WatchSource:0}: Error finding container f4985380d83311a4fe0ef175ea31de903dc68c6a4121250a4cc370dbb1ae3daa: Status 404 returned error can't find the container with id f4985380d83311a4fe0ef175ea31de903dc68c6a4121250a4cc370dbb1ae3daa Sep 10 17:20:14.024456 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:14.024422 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" event={"ID":"ccc1e89d-594c-41fd-a333-8a1ec6c061f8","Type":"ContainerStarted","Data":"f4985380d83311a4fe0ef175ea31de903dc68c6a4121250a4cc370dbb1ae3daa"} Sep 10 17:20:17.038151 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:17.038115 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" event={"ID":"ccc1e89d-594c-41fd-a333-8a1ec6c061f8","Type":"ContainerStarted","Data":"df374cbcf8f2b312b6f01f1f6a39f4fdd50f7d5196e5b6e07c63ea7c2ff07df9"} Sep 10 17:20:18.044053 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:18.043996 2557 generic.go:356] "Generic (PLEG): container finished" podID="ccc1e89d-594c-41fd-a333-8a1ec6c061f8" containerID="df374cbcf8f2b312b6f01f1f6a39f4fdd50f7d5196e5b6e07c63ea7c2ff07df9" exitCode=0 Sep 10 17:20:18.044436 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:18.044095 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" event={"ID":"ccc1e89d-594c-41fd-a333-8a1ec6c061f8","Type":"ContainerDied","Data":"df374cbcf8f2b312b6f01f1f6a39f4fdd50f7d5196e5b6e07c63ea7c2ff07df9"} Sep 10 17:20:19.049380 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:19.049351 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" event={"ID":"ccc1e89d-594c-41fd-a333-8a1ec6c061f8","Type":"ContainerStarted","Data":"353de86bab22527860b37cb6f5d324f578eff117b03b14a672ec9597dccee0d0"} Sep 10 17:20:19.049879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:19.049425 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:19.083979 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:19.083928 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" podStartSLOduration=2.909692785 podStartE2EDuration="7.08391284s" totalImagesPullingTime="3.170813362s" totalInitContainerRuntime="1.003406693s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:19.083191674 +0000 UTC m=+418.181090064" watchObservedRunningTime="2026-09-10 17:20:19.08391284 +0000 UTC m=+418.181811201" Sep 10 17:20:30.056141 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:30.056109 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8dc495f6d-twbtn" Sep 10 17:20:33.251965 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.251927 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j"] Sep 10 17:20:33.259513 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.259489 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.265809 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.265790 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 17:20:33.276007 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.275985 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 17:20:33.299722 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.299693 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j"] Sep 10 17:20:33.302884 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.302855 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7f6dae-5899-4d92-bbaa-902562678414-tls-certs\") pod \"kube-auth-proxy-555bd9596f-kmf2j\" (UID: \"bc7f6dae-5899-4d92-bbaa-902562678414\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.303007 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.302896 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc7f6dae-5899-4d92-bbaa-902562678414-tmp\") pod \"kube-auth-proxy-555bd9596f-kmf2j\" (UID: \"bc7f6dae-5899-4d92-bbaa-902562678414\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.303007 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.302922 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2nfc\" (UniqueName: \"kubernetes.io/projected/bc7f6dae-5899-4d92-bbaa-902562678414-kube-api-access-w2nfc\") pod \"kube-auth-proxy-555bd9596f-kmf2j\" (UID: \"bc7f6dae-5899-4d92-bbaa-902562678414\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.403643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.403623 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7f6dae-5899-4d92-bbaa-902562678414-tls-certs\") pod \"kube-auth-proxy-555bd9596f-kmf2j\" (UID: \"bc7f6dae-5899-4d92-bbaa-902562678414\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.403643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.403647 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc7f6dae-5899-4d92-bbaa-902562678414-tmp\") pod \"kube-auth-proxy-555bd9596f-kmf2j\" (UID: \"bc7f6dae-5899-4d92-bbaa-902562678414\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.403777 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.403665 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2nfc\" (UniqueName: \"kubernetes.io/projected/bc7f6dae-5899-4d92-bbaa-902562678414-kube-api-access-w2nfc\") pod \"kube-auth-proxy-555bd9596f-kmf2j\" (UID: \"bc7f6dae-5899-4d92-bbaa-902562678414\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.405988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.405960 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc7f6dae-5899-4d92-bbaa-902562678414-tmp\") pod \"kube-auth-proxy-555bd9596f-kmf2j\" (UID: \"bc7f6dae-5899-4d92-bbaa-902562678414\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.406126 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.406110 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc7f6dae-5899-4d92-bbaa-902562678414-tls-certs\") pod \"kube-auth-proxy-555bd9596f-kmf2j\" (UID: \"bc7f6dae-5899-4d92-bbaa-902562678414\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.510479 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.510420 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2nfc\" (UniqueName: \"kubernetes.io/projected/bc7f6dae-5899-4d92-bbaa-902562678414-kube-api-access-w2nfc\") pod \"kube-auth-proxy-555bd9596f-kmf2j\" (UID: \"bc7f6dae-5899-4d92-bbaa-902562678414\") " pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.568507 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.568484 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" Sep 10 17:20:33.900563 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:20:33.900534 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7f6dae_5899_4d92_bbaa_902562678414.slice/crio-b6aee424fe5d20eb2252699fd232329a6368a348e33e11376671acd5cc6c5f71 WatchSource:0}: Error finding container b6aee424fe5d20eb2252699fd232329a6368a348e33e11376671acd5cc6c5f71: Status 404 returned error can't find the container with id b6aee424fe5d20eb2252699fd232329a6368a348e33e11376671acd5cc6c5f71 Sep 10 17:20:33.900682 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:33.900592 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j"] Sep 10 17:20:34.102733 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:34.102706 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" event={"ID":"bc7f6dae-5899-4d92-bbaa-902562678414","Type":"ContainerStarted","Data":"b6aee424fe5d20eb2252699fd232329a6368a348e33e11376671acd5cc6c5f71"} Sep 10 17:20:37.115577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:37.115544 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" event={"ID":"bc7f6dae-5899-4d92-bbaa-902562678414","Type":"ContainerStarted","Data":"18a8b2ef2c541426d044e4fb577dd9a728c6e963b70a5bfe82cf8b6ad1cb1c3d"} Sep 10 17:20:39.424892 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.424837 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-555bd9596f-kmf2j" podStartSLOduration=3.705698157 podStartE2EDuration="6.42482195s" totalImagesPullingTime="2.719123793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:37.234204483 +0000 UTC m=+436.332102843" watchObservedRunningTime="2026-09-10 17:20:39.42482195 +0000 UTC m=+438.522720314" Sep 10 17:20:39.425479 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.425457 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb"] Sep 10 17:20:39.427604 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.427582 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.445891 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.445873 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 17:20:39.446402 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.446386 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 17:20:39.446579 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.446563 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 17:20:39.447173 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.447158 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-96nhf\"" Sep 10 17:20:39.448181 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.448162 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 17:20:39.487741 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.487720 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb"] Sep 10 17:20:39.546447 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.546426 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492mn\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-kube-api-access-492mn\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.546552 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.546460 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.546552 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.546480 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.546552 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.546498 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.546675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.546587 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.546675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.546615 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.546675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.546657 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.647583 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.647559 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492mn\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-kube-api-access-492mn\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.647687 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.647590 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.647687 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.647607 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.647687 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.647625 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.647687 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.647649 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.647881 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.647704 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.647881 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.647767 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.648294 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.648267 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.649945 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.649923 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.650076 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.650061 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.650339 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.650321 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.650464 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.650445 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.682400 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.682346 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.684944 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.684927 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-492mn\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-kube-api-access-492mn\") pod \"istiod-openshift-gateway-7b7995d5cd-mw2nb\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:39.735965 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:39.735948 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:40.063613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:40.063589 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb"] Sep 10 17:20:40.065047 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:20:40.065003 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7a2e3c_da6e_49fa_81aa_ecf42c7c7e18.slice/crio-b6de6ba842dc39b4cc0166174dff97cd613f06c79bd0bf553c684d9fb668649a WatchSource:0}: Error finding container b6de6ba842dc39b4cc0166174dff97cd613f06c79bd0bf553c684d9fb668649a: Status 404 returned error can't find the container with id b6de6ba842dc39b4cc0166174dff97cd613f06c79bd0bf553c684d9fb668649a Sep 10 17:20:40.127535 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:40.127512 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" event={"ID":"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18","Type":"ContainerStarted","Data":"b6de6ba842dc39b4cc0166174dff97cd613f06c79bd0bf553c684d9fb668649a"} Sep 10 17:20:42.723240 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:42.723189 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:20:42.723521 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:42.723271 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:20:43.142758 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:43.142722 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" event={"ID":"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18","Type":"ContainerStarted","Data":"fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943"} Sep 10 17:20:43.142951 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:43.142790 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:43.144456 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:43.144422 2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-mw2nb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 17:20:43.144558 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:43.144474 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" podUID="ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 17:20:43.196451 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:43.196405 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" podStartSLOduration=1.540474347 podStartE2EDuration="4.196248501s" totalImagesPullingTime="2.655774154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:43.190692432 +0000 UTC m=+442.288590793" watchObservedRunningTime="2026-09-10 17:20:43.196248501 +0000 UTC m=+442.294146862" Sep 10 17:20:44.147745 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:44.147675 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:54.712314 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.712277 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx"] Sep 10 17:20:54.718674 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.718651 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.773819 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.773780 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.773819 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.773816 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.774023 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.773840 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a79dedd-0eae-4a3c-a898-5d1c0b331859-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.774023 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.773918 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.774023 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.773966 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a79dedd-0eae-4a3c-a898-5d1c0b331859-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.774023 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.773991 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.774187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.774026 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpdg\" (UniqueName: \"kubernetes.io/projected/6a79dedd-0eae-4a3c-a898-5d1c0b331859-kube-api-access-5rpdg\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.811741 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.811710 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx"] Sep 10 17:20:54.874592 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.874558 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.874795 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.874607 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a79dedd-0eae-4a3c-a898-5d1c0b331859-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.874795 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.874631 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.874795 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.874648 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpdg\" (UniqueName: \"kubernetes.io/projected/6a79dedd-0eae-4a3c-a898-5d1c0b331859-kube-api-access-5rpdg\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.874795 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.874678 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.874795 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.874696 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.874795 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.874722 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a79dedd-0eae-4a3c-a898-5d1c0b331859-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.875501 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.875465 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.877022 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.876996 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a79dedd-0eae-4a3c-a898-5d1c0b331859-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.877163 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.877134 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.877506 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.877484 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6a79dedd-0eae-4a3c-a898-5d1c0b331859-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.877555 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.877484 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.886332 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.886304 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a79dedd-0eae-4a3c-a898-5d1c0b331859-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:54.892305 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:54.892276 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpdg\" (UniqueName: \"kubernetes.io/projected/6a79dedd-0eae-4a3c-a898-5d1c0b331859-kube-api-access-5rpdg\") pod \"istiod-openshift-gateway-5677ccb49b-thzdx\" (UID: \"6a79dedd-0eae-4a3c-a898-5d1c0b331859\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:55.026756 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:55.026721 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:55.360197 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:55.360119 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx"] Sep 10 17:20:55.363994 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:20:55.363969 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a79dedd_0eae_4a3c_a898_5d1c0b331859.slice/crio-7e5b2886a067f2d3e9c182b33c9bbb3d2ab800a3e9dcab7cd2cf4c749141dc03 WatchSource:0}: Error finding container 7e5b2886a067f2d3e9c182b33c9bbb3d2ab800a3e9dcab7cd2cf4c749141dc03: Status 404 returned error can't find the container with id 7e5b2886a067f2d3e9c182b33c9bbb3d2ab800a3e9dcab7cd2cf4c749141dc03 Sep 10 17:20:55.365993 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:55.365959 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:20:55.366121 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:55.366065 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:20:56.195210 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:56.195173 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" event={"ID":"6a79dedd-0eae-4a3c-a898-5d1c0b331859","Type":"ContainerStarted","Data":"2ba82a20489dc30afffe0c15b866d0e795ab0da46eb11a910b7a333e5322a0b5"} Sep 10 17:20:56.195650 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:56.195235 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" event={"ID":"6a79dedd-0eae-4a3c-a898-5d1c0b331859","Type":"ContainerStarted","Data":"7e5b2886a067f2d3e9c182b33c9bbb3d2ab800a3e9dcab7cd2cf4c749141dc03"} Sep 10 17:20:56.195650 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:56.195403 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:56.197049 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:56.197006 2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-thzdx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 17:20:56.197250 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:56.197227 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" podUID="6a79dedd-0eae-4a3c-a898-5d1c0b331859" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 17:20:56.279787 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:56.279735 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" podStartSLOduration=2.279718511 podStartE2EDuration="2.279718511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:56.278858446 +0000 UTC m=+455.376756807" watchObservedRunningTime="2026-09-10 17:20:56.279718511 +0000 UTC m=+455.377616871" Sep 10 17:20:57.200282 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.200251 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-thzdx" Sep 10 17:20:57.436437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.436403 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb"] Sep 10 17:20:57.436639 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.436598 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" podUID="ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" containerName="discovery" containerID="cri-o://fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943" gracePeriod=30 Sep 10 17:20:57.680845 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.680820 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:57.798822 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.798792 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-ca-configmap\") pod \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " Sep 10 17:20:57.798976 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.798832 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-dns-cert\") pod \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " Sep 10 17:20:57.798976 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.798853 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-492mn\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-kube-api-access-492mn\") pod \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " Sep 10 17:20:57.798976 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.798887 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-token\") pod \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " Sep 10 17:20:57.799178 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.798988 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-cacerts\") pod \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " Sep 10 17:20:57.799178 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.799062 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-kubeconfig\") pod \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " Sep 10 17:20:57.799178 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.799108 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-local-certs\") pod \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\" (UID: \"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18\") " Sep 10 17:20:57.799330 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.799261 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" (UID: "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:20:57.801347 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.801317 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" (UID: "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:20:57.801474 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.801391 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-cacerts" (OuterVolumeSpecName: "cacerts") pod "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" (UID: "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:20:57.801543 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.801522 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" (UID: "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:20:57.801543 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.801536 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" (UID: "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:20:57.801612 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.801549 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-kube-api-access-492mn" (OuterVolumeSpecName: "kube-api-access-492mn") pod "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" (UID: "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18"). InnerVolumeSpecName "kube-api-access-492mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:20:57.801646 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.801604 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-local-certs" (OuterVolumeSpecName: "local-certs") pod "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" (UID: "ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:20:57.899679 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.899638 2557 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-cacerts\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:20:57.899679 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.899673 2557 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-kubeconfig\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:20:57.899679 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.899684 2557 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-local-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:20:57.899679 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.899692 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-ca-configmap\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:20:57.899915 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.899701 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-csr-dns-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:20:57.899915 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.899709 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-492mn\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-kube-api-access-492mn\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:20:57.899915 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:57.899718 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:20:58.204781 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:58.204695 2557 generic.go:356] "Generic (PLEG): container finished" podID="ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" containerID="fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943" exitCode=0 Sep 10 17:20:58.204781 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:58.204738 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" Sep 10 17:20:58.204781 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:58.204772 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" event={"ID":"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18","Type":"ContainerDied","Data":"fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943"} Sep 10 17:20:58.205322 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:58.204826 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb" event={"ID":"ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18","Type":"ContainerDied","Data":"b6de6ba842dc39b4cc0166174dff97cd613f06c79bd0bf553c684d9fb668649a"} Sep 10 17:20:58.205322 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:58.204878 2557 scope.go:117] "RemoveContainer" containerID="fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943" Sep 10 17:20:58.213733 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:58.213705 2557 scope.go:117] "RemoveContainer" containerID="fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943" Sep 10 17:20:58.213990 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:20:58.213971 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943\": container with ID starting with fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943 not found: ID does not exist" containerID="fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943" Sep 10 17:20:58.214063 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:58.214002 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943"} err="failed to get container status \"fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943\": rpc error: code = NotFound desc = could not find container \"fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943\": container with ID starting with fff0a7e12eddb3d4c1709bc2ad139a7f57098ef2208dfd8f14a351432b203943 not found: ID does not exist" Sep 10 17:20:58.249407 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:58.249382 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb"] Sep 10 17:20:58.257742 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:58.257714 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mw2nb"] Sep 10 17:20:59.385125 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.385050 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" path="/var/lib/kubelet/pods/ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18/volumes" Sep 10 17:20:59.821890 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.821861 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gctlv"] Sep 10 17:20:59.822168 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.822155 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" containerName="discovery" Sep 10 17:20:59.822224 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.822171 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" containerName="discovery" Sep 10 17:20:59.822224 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.822222 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed7a2e3c-da6e-49fa-81aa-ecf42c7c7e18" containerName="discovery" Sep 10 17:20:59.824447 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.824419 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:20:59.829390 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:20:59.829357 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kserve-controller-manager-d6f4b65b8-gctlv\" is forbidden: User \"system:node:ip-10-0-133-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-191.ec2.internal' and this object" podUID="0847ed17-5118-45dc-b679-81af2c15a17c" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:20:59.829514 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:20:59.829376 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-controller-manager-dockercfg-h46ph\" is forbidden: User \"system:node:ip-10-0-133-191.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-191.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-h46ph\"" type="*v1.Secret" Sep 10 17:20:59.829514 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:20:59.829444 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-133-191.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-191.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" Sep 10 17:20:59.850700 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.850669 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv"] Sep 10 17:20:59.852463 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.852449 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:20:59.860192 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.860173 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 17:20:59.860729 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.860708 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-c2gnb\"" Sep 10 17:20:59.886584 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.886557 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gctlv"] Sep 10 17:20:59.913950 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.913922 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljfr\" (UniqueName: \"kubernetes.io/projected/ca796d01-4160-41df-a46b-20fdfea7324a-kube-api-access-nljfr\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hkv\" (UID: \"ca796d01-4160-41df-a46b-20fdfea7324a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:20:59.914062 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.913966 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0847ed17-5118-45dc-b679-81af2c15a17c-cert\") pod \"kserve-controller-manager-d6f4b65b8-gctlv\" (UID: \"0847ed17-5118-45dc-b679-81af2c15a17c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:20:59.914062 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.914050 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6gk\" (UniqueName: \"kubernetes.io/projected/0847ed17-5118-45dc-b679-81af2c15a17c-kube-api-access-5z6gk\") pod \"kserve-controller-manager-d6f4b65b8-gctlv\" (UID: \"0847ed17-5118-45dc-b679-81af2c15a17c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:20:59.914137 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.914075 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca796d01-4160-41df-a46b-20fdfea7324a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hkv\" (UID: \"ca796d01-4160-41df-a46b-20fdfea7324a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:20:59.942775 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:20:59.942750 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv"] Sep 10 17:21:00.014797 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.014767 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6gk\" (UniqueName: \"kubernetes.io/projected/0847ed17-5118-45dc-b679-81af2c15a17c-kube-api-access-5z6gk\") pod \"kserve-controller-manager-d6f4b65b8-gctlv\" (UID: \"0847ed17-5118-45dc-b679-81af2c15a17c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:21:00.014797 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.014799 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca796d01-4160-41df-a46b-20fdfea7324a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hkv\" (UID: \"ca796d01-4160-41df-a46b-20fdfea7324a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:21:00.014991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.014829 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nljfr\" (UniqueName: \"kubernetes.io/projected/ca796d01-4160-41df-a46b-20fdfea7324a-kube-api-access-nljfr\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hkv\" (UID: \"ca796d01-4160-41df-a46b-20fdfea7324a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:21:00.014991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.014902 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0847ed17-5118-45dc-b679-81af2c15a17c-cert\") pod \"kserve-controller-manager-d6f4b65b8-gctlv\" (UID: \"0847ed17-5118-45dc-b679-81af2c15a17c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:21:00.017294 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.017271 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca796d01-4160-41df-a46b-20fdfea7324a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hkv\" (UID: \"ca796d01-4160-41df-a46b-20fdfea7324a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:21:00.041205 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.041179 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljfr\" (UniqueName: \"kubernetes.io/projected/ca796d01-4160-41df-a46b-20fdfea7324a-kube-api-access-nljfr\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hkv\" (UID: \"ca796d01-4160-41df-a46b-20fdfea7324a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:21:00.055265 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.055244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6gk\" (UniqueName: \"kubernetes.io/projected/0847ed17-5118-45dc-b679-81af2c15a17c-kube-api-access-5z6gk\") pod \"kserve-controller-manager-d6f4b65b8-gctlv\" (UID: \"0847ed17-5118-45dc-b679-81af2c15a17c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:21:00.161576 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.161493 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:21:00.524297 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.524265 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv"] Sep 10 17:21:00.538026 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:21:00.537994 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podca796d01_4160_41df_a46b_20fdfea7324a.slice/crio-88846ff73b34c6f7fe594ecc5e931a50c7316997b62c6522a368a12965f3120f WatchSource:0}: Error finding container 88846ff73b34c6f7fe594ecc5e931a50c7316997b62c6522a368a12965f3120f: Status 404 returned error can't find the container with id 88846ff73b34c6f7fe594ecc5e931a50c7316997b62c6522a368a12965f3120f Sep 10 17:21:00.890466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.890390 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 17:21:00.897642 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:00.897619 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0847ed17-5118-45dc-b679-81af2c15a17c-cert\") pod \"kserve-controller-manager-d6f4b65b8-gctlv\" (UID: \"0847ed17-5118-45dc-b679-81af2c15a17c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:21:01.124909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:01.124878 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-h46ph\"" Sep 10 17:21:01.134658 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:01.134634 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:21:01.220992 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:01.220961 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" event={"ID":"ca796d01-4160-41df-a46b-20fdfea7324a","Type":"ContainerStarted","Data":"88846ff73b34c6f7fe594ecc5e931a50c7316997b62c6522a368a12965f3120f"} Sep 10 17:21:01.432022 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:01.431998 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gctlv"] Sep 10 17:21:01.514061 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:21:01.514009 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0847ed17_5118_45dc_b679_81af2c15a17c.slice/crio-a18ec2dc4f569148bce993f7efa684917a672b44f613b6b84b6fa2e118040d10 WatchSource:0}: Error finding container a18ec2dc4f569148bce993f7efa684917a672b44f613b6b84b6fa2e118040d10: Status 404 returned error can't find the container with id a18ec2dc4f569148bce993f7efa684917a672b44f613b6b84b6fa2e118040d10 Sep 10 17:21:02.228533 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:02.228496 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" event={"ID":"0847ed17-5118-45dc-b679-81af2c15a17c","Type":"ContainerStarted","Data":"a18ec2dc4f569148bce993f7efa684917a672b44f613b6b84b6fa2e118040d10"} Sep 10 17:21:04.239702 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:04.239613 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" event={"ID":"0847ed17-5118-45dc-b679-81af2c15a17c","Type":"ContainerStarted","Data":"dec19f8f862ced29defa28d97af97ab0d7139519b2b45f4f60ad5fac1d262720"} Sep 10 17:21:04.240154 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:04.239713 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:21:04.241274 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:04.241253 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" event={"ID":"ca796d01-4160-41df-a46b-20fdfea7324a","Type":"ContainerStarted","Data":"a6f493110dc26de2f9e25c7fad7eacda3e395f5418403af0a683afc44e167213"} Sep 10 17:21:04.241386 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:04.241379 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:21:04.274934 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:04.274893 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" podStartSLOduration=2.874821872 podStartE2EDuration="5.274879807s" totalImagesPullingTime="2.400057935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:21:04.272044348 +0000 UTC m=+463.369942701" watchObservedRunningTime="2026-09-10 17:21:04.274879807 +0000 UTC m=+463.372778167" Sep 10 17:21:04.316796 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:04.316753 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" podStartSLOduration=1.940023692 podStartE2EDuration="5.316741607s" totalImagesPullingTime="3.376717915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:21:04.315047081 +0000 UTC m=+463.412945436" watchObservedRunningTime="2026-09-10 17:21:04.316741607 +0000 UTC m=+463.414639967" Sep 10 17:21:35.247062 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:35.247018 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hkv" Sep 10 17:21:35.250234 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:21:35.250213 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gctlv" Sep 10 17:22:21.656077 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.655984 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc"] Sep 10 17:22:21.657960 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.657943 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" Sep 10 17:22:21.663898 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.663873 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 10 17:22:21.664101 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.664083 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 17:22:21.664208 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.664168 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q54qw\"" Sep 10 17:22:21.664451 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.664436 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 17:22:21.680226 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.680202 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc"] Sep 10 17:22:21.692080 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.692056 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv54t\" (UniqueName: \"kubernetes.io/projected/a5d4e00e-0ab2-4aa2-8201-9db87b49f957-kube-api-access-zv54t\") pod \"dns-operator-controller-manager-7b84cd7c9c-g9qvc\" (UID: \"a5d4e00e-0ab2-4aa2-8201-9db87b49f957\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" Sep 10 17:22:21.793115 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.793083 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv54t\" (UniqueName: \"kubernetes.io/projected/a5d4e00e-0ab2-4aa2-8201-9db87b49f957-kube-api-access-zv54t\") pod \"dns-operator-controller-manager-7b84cd7c9c-g9qvc\" (UID: \"a5d4e00e-0ab2-4aa2-8201-9db87b49f957\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" Sep 10 17:22:21.810576 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.810549 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv54t\" (UniqueName: \"kubernetes.io/projected/a5d4e00e-0ab2-4aa2-8201-9db87b49f957-kube-api-access-zv54t\") pod \"dns-operator-controller-manager-7b84cd7c9c-g9qvc\" (UID: \"a5d4e00e-0ab2-4aa2-8201-9db87b49f957\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" Sep 10 17:22:21.968502 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:21.968420 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" Sep 10 17:22:22.308369 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:22.308336 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc"] Sep 10 17:22:22.311598 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:22:22.311571 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d4e00e_0ab2_4aa2_8201_9db87b49f957.slice/crio-e937c857c67c0e262fc45b60dc84449fb5b252cafc883200e7127be05aa31dab WatchSource:0}: Error finding container e937c857c67c0e262fc45b60dc84449fb5b252cafc883200e7127be05aa31dab: Status 404 returned error can't find the container with id e937c857c67c0e262fc45b60dc84449fb5b252cafc883200e7127be05aa31dab Sep 10 17:22:22.548717 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:22.548683 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" event={"ID":"a5d4e00e-0ab2-4aa2-8201-9db87b49f957","Type":"ContainerStarted","Data":"e937c857c67c0e262fc45b60dc84449fb5b252cafc883200e7127be05aa31dab"} Sep 10 17:22:23.867617 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:23.867580 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd"] Sep 10 17:22:23.869697 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:23.869676 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" Sep 10 17:22:23.873210 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:23.873180 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-cmhc6\"" Sep 10 17:22:23.890312 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:23.890289 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd"] Sep 10 17:22:23.910068 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:23.910041 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/049d05ed-f166-4981-8474-fb1170d5a0f7-kube-api-access-v8phf\") pod \"limitador-operator-controller-manager-74495c69f8-hzwrd\" (UID: \"049d05ed-f166-4981-8474-fb1170d5a0f7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" Sep 10 17:22:24.010889 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:24.010853 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/049d05ed-f166-4981-8474-fb1170d5a0f7-kube-api-access-v8phf\") pod \"limitador-operator-controller-manager-74495c69f8-hzwrd\" (UID: \"049d05ed-f166-4981-8474-fb1170d5a0f7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" Sep 10 17:22:24.025641 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:24.025610 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/049d05ed-f166-4981-8474-fb1170d5a0f7-kube-api-access-v8phf\") pod \"limitador-operator-controller-manager-74495c69f8-hzwrd\" (UID: \"049d05ed-f166-4981-8474-fb1170d5a0f7\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" Sep 10 17:22:24.181767 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:24.181680 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" Sep 10 17:22:25.205791 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:25.205714 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd"] Sep 10 17:22:25.209821 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:22:25.209793 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049d05ed_f166_4981_8474_fb1170d5a0f7.slice/crio-bb1ec0e2ae64df16b964943542818e971a0597894ab9ec41af6cfef93a2c1e8c WatchSource:0}: Error finding container bb1ec0e2ae64df16b964943542818e971a0597894ab9ec41af6cfef93a2c1e8c: Status 404 returned error can't find the container with id bb1ec0e2ae64df16b964943542818e971a0597894ab9ec41af6cfef93a2c1e8c Sep 10 17:22:25.563224 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:25.563189 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" event={"ID":"a5d4e00e-0ab2-4aa2-8201-9db87b49f957","Type":"ContainerStarted","Data":"bceb793d87dd1073bf790a23a76598ae3027ee4b11297dab18463b0cc9442817"} Sep 10 17:22:25.563430 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:25.563270 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" Sep 10 17:22:25.564521 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:25.564501 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" event={"ID":"049d05ed-f166-4981-8474-fb1170d5a0f7","Type":"ContainerStarted","Data":"bb1ec0e2ae64df16b964943542818e971a0597894ab9ec41af6cfef93a2c1e8c"} Sep 10 17:22:25.596245 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:25.596192 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" podStartSLOduration=1.9698026720000001 podStartE2EDuration="4.596174789s" totalImagesPullingTime="2.626372117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:22:25.595417691 +0000 UTC m=+544.693316053" watchObservedRunningTime="2026-09-10 17:22:25.596174789 +0000 UTC m=+544.694073149" Sep 10 17:22:27.575433 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:27.575397 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" event={"ID":"049d05ed-f166-4981-8474-fb1170d5a0f7","Type":"ContainerStarted","Data":"c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0"} Sep 10 17:22:27.575879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:27.575519 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" Sep 10 17:22:27.618141 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:27.618090 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" podStartSLOduration=2.917129607 podStartE2EDuration="4.61807498s" totalImagesPullingTime="1.700945373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:22:27.615681914 +0000 UTC m=+546.713580274" watchObservedRunningTime="2026-09-10 17:22:27.61807498 +0000 UTC m=+546.715973340" Sep 10 17:22:36.573109 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:36.573074 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g9qvc" Sep 10 17:22:38.583498 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:38.583461 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" Sep 10 17:22:47.261232 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.261199 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd"] Sep 10 17:22:47.261589 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.261430 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" podUID="049d05ed-f166-4981-8474-fb1170d5a0f7" containerName="manager" containerID="cri-o://c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0" gracePeriod=2 Sep 10 17:22:47.265875 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.265846 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd"] Sep 10 17:22:47.387277 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.387247 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld"] Sep 10 17:22:47.387526 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.387514 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="049d05ed-f166-4981-8474-fb1170d5a0f7" containerName="manager" Sep 10 17:22:47.387587 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.387528 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="049d05ed-f166-4981-8474-fb1170d5a0f7" containerName="manager" Sep 10 17:22:47.387626 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.387590 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="049d05ed-f166-4981-8474-fb1170d5a0f7" containerName="manager" Sep 10 17:22:47.389278 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.389255 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:47.392277 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.392258 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fpvh4\"" Sep 10 17:22:47.396797 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:22:47.396772 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-f62ld\" is forbidden: User \"system:node:ip-10-0-133-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-191.ec2.internal' and this object" podUID="b1f0776c-587c-48d4-8a3c-3b9dacb18781" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:47.463766 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.463741 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld"] Sep 10 17:22:47.482283 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.482251 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f0776c-587c-48d4-8a3c-3b9dacb18781-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f62ld\" (UID: \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:47.482406 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.482288 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hgqn\" (UniqueName: \"kubernetes.io/projected/b1f0776c-587c-48d4-8a3c-3b9dacb18781-kube-api-access-9hgqn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f62ld\" (UID: \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:47.514454 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.514405 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" Sep 10 17:22:47.583105 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.583083 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/049d05ed-f166-4981-8474-fb1170d5a0f7-kube-api-access-v8phf\") pod \"049d05ed-f166-4981-8474-fb1170d5a0f7\" (UID: \"049d05ed-f166-4981-8474-fb1170d5a0f7\") " Sep 10 17:22:47.583238 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.583221 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f0776c-587c-48d4-8a3c-3b9dacb18781-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f62ld\" (UID: \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:47.583277 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.583255 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hgqn\" (UniqueName: \"kubernetes.io/projected/b1f0776c-587c-48d4-8a3c-3b9dacb18781-kube-api-access-9hgqn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f62ld\" (UID: \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:47.583623 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.583607 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f0776c-587c-48d4-8a3c-3b9dacb18781-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f62ld\" (UID: \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:47.585137 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.585121 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049d05ed-f166-4981-8474-fb1170d5a0f7-kube-api-access-v8phf" (OuterVolumeSpecName: "kube-api-access-v8phf") pod "049d05ed-f166-4981-8474-fb1170d5a0f7" (UID: "049d05ed-f166-4981-8474-fb1170d5a0f7"). InnerVolumeSpecName "kube-api-access-v8phf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:22:47.624004 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.623975 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hgqn\" (UniqueName: \"kubernetes.io/projected/b1f0776c-587c-48d4-8a3c-3b9dacb18781-kube-api-access-9hgqn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f62ld\" (UID: \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:47.662980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.662956 2557 generic.go:356] "Generic (PLEG): container finished" podID="049d05ed-f166-4981-8474-fb1170d5a0f7" containerID="c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0" exitCode=0 Sep 10 17:22:47.663109 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.662995 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hzwrd" Sep 10 17:22:47.663109 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.663080 2557 scope.go:117] "RemoveContainer" containerID="c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0" Sep 10 17:22:47.671748 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.671731 2557 scope.go:117] "RemoveContainer" containerID="c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0" Sep 10 17:22:47.672017 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:22:47.671995 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0\": container with ID starting with c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0 not found: ID does not exist" containerID="c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0" Sep 10 17:22:47.672099 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.672025 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0"} err="failed to get container status \"c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0\": rpc error: code = NotFound desc = could not find container \"c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0\": container with ID starting with c957cc0d45589df0d0844697ad54cb4cd9ab8dedc79071c1d6f880e3d2fa85b0 not found: ID does not exist" Sep 10 17:22:47.683662 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.683640 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8phf\" (UniqueName: \"kubernetes.io/projected/049d05ed-f166-4981-8474-fb1170d5a0f7-kube-api-access-v8phf\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:22:47.700604 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:47.700582 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:48.009125 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:22:48.009094 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f0776c_587c_48d4_8a3c_3b9dacb18781.slice/crio-38b28e7695bad740fff14c2d8921a57bba1d16ca48acd7f95d3f12699f6c6608 WatchSource:0}: Error finding container 38b28e7695bad740fff14c2d8921a57bba1d16ca48acd7f95d3f12699f6c6608: Status 404 returned error can't find the container with id 38b28e7695bad740fff14c2d8921a57bba1d16ca48acd7f95d3f12699f6c6608 Sep 10 17:22:48.027029 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:48.027005 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld"] Sep 10 17:22:48.670862 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:48.670821 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" event={"ID":"b1f0776c-587c-48d4-8a3c-3b9dacb18781","Type":"ContainerStarted","Data":"38b28e7695bad740fff14c2d8921a57bba1d16ca48acd7f95d3f12699f6c6608"} Sep 10 17:22:49.387135 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:49.387095 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="049d05ed-f166-4981-8474-fb1170d5a0f7" path="/var/lib/kubelet/pods/049d05ed-f166-4981-8474-fb1170d5a0f7/volumes" Sep 10 17:22:51.687159 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:51.687074 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" event={"ID":"b1f0776c-587c-48d4-8a3c-3b9dacb18781","Type":"ContainerStarted","Data":"5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a"} Sep 10 17:22:51.687514 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:51.687196 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:22:51.732372 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:22:51.732326 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" podStartSLOduration=1.3066602600000001 podStartE2EDuration="4.732312168s" totalImagesPullingTime="3.425651908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:22:51.731304015 +0000 UTC m=+570.829202390" watchObservedRunningTime="2026-09-10 17:22:51.732312168 +0000 UTC m=+570.830210529" Sep 10 17:23:02.693727 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:02.693685 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:23:02.873831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:02.873797 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld"] Sep 10 17:23:02.874055 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:02.873991 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" podUID="b1f0776c-587c-48d4-8a3c-3b9dacb18781" containerName="manager" containerID="cri-o://5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a" gracePeriod=10 Sep 10 17:23:03.112725 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.112703 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:23:03.203092 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.203057 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f0776c-587c-48d4-8a3c-3b9dacb18781-extensions-socket-volume\") pod \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\" (UID: \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\") " Sep 10 17:23:03.203268 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.203128 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hgqn\" (UniqueName: \"kubernetes.io/projected/b1f0776c-587c-48d4-8a3c-3b9dacb18781-kube-api-access-9hgqn\") pod \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\" (UID: \"b1f0776c-587c-48d4-8a3c-3b9dacb18781\") " Sep 10 17:23:03.203557 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.203523 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f0776c-587c-48d4-8a3c-3b9dacb18781-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b1f0776c-587c-48d4-8a3c-3b9dacb18781" (UID: "b1f0776c-587c-48d4-8a3c-3b9dacb18781"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:23:03.205043 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.205008 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f0776c-587c-48d4-8a3c-3b9dacb18781-kube-api-access-9hgqn" (OuterVolumeSpecName: "kube-api-access-9hgqn") pod "b1f0776c-587c-48d4-8a3c-3b9dacb18781" (UID: "b1f0776c-587c-48d4-8a3c-3b9dacb18781"). InnerVolumeSpecName "kube-api-access-9hgqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:23:03.303897 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.303869 2557 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f0776c-587c-48d4-8a3c-3b9dacb18781-extensions-socket-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:23:03.303897 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.303893 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hgqn\" (UniqueName: \"kubernetes.io/projected/b1f0776c-587c-48d4-8a3c-3b9dacb18781-kube-api-access-9hgqn\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:23:03.738020 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.737934 2557 generic.go:356] "Generic (PLEG): container finished" podID="b1f0776c-587c-48d4-8a3c-3b9dacb18781" containerID="5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a" exitCode=0 Sep 10 17:23:03.738020 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.738004 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" Sep 10 17:23:03.738444 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.738011 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" event={"ID":"b1f0776c-587c-48d4-8a3c-3b9dacb18781","Type":"ContainerDied","Data":"5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a"} Sep 10 17:23:03.738444 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.738136 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld" event={"ID":"b1f0776c-587c-48d4-8a3c-3b9dacb18781","Type":"ContainerDied","Data":"38b28e7695bad740fff14c2d8921a57bba1d16ca48acd7f95d3f12699f6c6608"} Sep 10 17:23:03.738444 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.738159 2557 scope.go:117] "RemoveContainer" containerID="5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a" Sep 10 17:23:03.746533 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.746518 2557 scope.go:117] "RemoveContainer" containerID="5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a" Sep 10 17:23:03.746788 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:23:03.746768 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a\": container with ID starting with 5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a not found: ID does not exist" containerID="5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a" Sep 10 17:23:03.746848 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.746795 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a"} err="failed to get container status \"5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a\": rpc error: code = NotFound desc = could not find container \"5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a\": container with ID starting with 5ab6d0951c177a782d25a5b9397c5610aac4ef15dc384f5b00c4e26b5e5fa50a not found: ID does not exist" Sep 10 17:23:03.794373 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.794346 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld"] Sep 10 17:23:03.812012 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:03.811981 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f62ld"] Sep 10 17:23:05.385846 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:05.385813 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f0776c-587c-48d4-8a3c-3b9dacb18781" path="/var/lib/kubelet/pods/b1f0776c-587c-48d4-8a3c-3b9dacb18781/volumes" Sep 10 17:23:19.494577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.494540 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9"] Sep 10 17:23:19.495071 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.494845 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f0776c-587c-48d4-8a3c-3b9dacb18781" containerName="manager" Sep 10 17:23:19.495071 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.494857 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f0776c-587c-48d4-8a3c-3b9dacb18781" containerName="manager" Sep 10 17:23:19.495071 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.494922 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f0776c-587c-48d4-8a3c-3b9dacb18781" containerName="manager" Sep 10 17:23:19.497685 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.497666 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.500515 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.500497 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gvcfh\"" Sep 10 17:23:19.513978 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.513955 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9"] Sep 10 17:23:19.524198 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.524176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0578c7d9-e911-4d2d-9266-7862658d07f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.524303 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.524212 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.524303 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.524232 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.524303 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.524268 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnhx\" (UniqueName: \"kubernetes.io/projected/0578c7d9-e911-4d2d-9266-7862658d07f4-kube-api-access-9qnhx\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.524437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.524319 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.524437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.524337 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.524437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.524369 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.524437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.524405 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.524437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.524431 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625104 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625314 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625146 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625314 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625188 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0578c7d9-e911-4d2d-9266-7862658d07f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625314 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625227 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625314 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625253 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625314 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625278 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnhx\" (UniqueName: \"kubernetes.io/projected/0578c7d9-e911-4d2d-9266-7862658d07f4-kube-api-access-9qnhx\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625635 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625330 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625635 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625353 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625635 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625614 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625524 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625637 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625757 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625947 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625842 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.625947 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.625935 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0578c7d9-e911-4d2d-9266-7862658d07f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.627586 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.627564 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.627687 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.627653 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.636461 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.636442 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnhx\" (UniqueName: \"kubernetes.io/projected/0578c7d9-e911-4d2d-9266-7862658d07f4-kube-api-access-9qnhx\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.637943 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.637925 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0578c7d9-e911-4d2d-9266-7862658d07f4-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-8k8f9\" (UID: \"0578c7d9-e911-4d2d-9266-7862658d07f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:19.807888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:19.807807 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:20.088955 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:20.088925 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9"] Sep 10 17:23:20.089993 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:23:20.089960 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0578c7d9_e911_4d2d_9266_7862658d07f4.slice/crio-42d622fa85e31d90c1357b3f6cd9f40164aa6a7e912b03e2d1c487fd0beea6e3 WatchSource:0}: Error finding container 42d622fa85e31d90c1357b3f6cd9f40164aa6a7e912b03e2d1c487fd0beea6e3: Status 404 returned error can't find the container with id 42d622fa85e31d90c1357b3f6cd9f40164aa6a7e912b03e2d1c487fd0beea6e3 Sep 10 17:23:20.807346 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:20.807309 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" event={"ID":"0578c7d9-e911-4d2d-9266-7862658d07f4","Type":"ContainerStarted","Data":"42d622fa85e31d90c1357b3f6cd9f40164aa6a7e912b03e2d1c487fd0beea6e3"} Sep 10 17:23:21.298484 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:21.298461 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:23:21.298863 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:21.298842 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:23:23.378418 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:23.378375 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:23:23.378717 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:23.378452 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:23:23.378717 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:23.378480 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:23:23.821350 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:23.821307 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" event={"ID":"0578c7d9-e911-4d2d-9266-7862658d07f4","Type":"ContainerStarted","Data":"18727ab7b87bd7d7b315fb1928a8960ab5d52fa0aa0b99bf1c742abe2560a466"} Sep 10 17:23:23.848331 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:23.848269 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" podStartSLOduration=1.562057931 podStartE2EDuration="4.848249512s" totalImagesPullingTime="3.286191581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:23:23.847499339 +0000 UTC m=+602.945397701" watchObservedRunningTime="2026-09-10 17:23:23.848249512 +0000 UTC m=+602.946147873" Sep 10 17:23:24.808886 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:24.808845 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:24.813596 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:24.813570 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:24.824991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:24.824969 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:24.825644 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:24.825626 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:23:25.033841 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.033805 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r4z9t"] Sep 10 17:23:25.043846 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.043823 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:25.046950 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.046935 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fs6h2\"" Sep 10 17:23:25.047388 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.047371 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 17:23:25.057097 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.057077 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r4z9t"] Sep 10 17:23:25.074304 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.074247 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-config-file\") pod \"limitador-limitador-8588749977-r4z9t\" (UID: \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\") " pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:25.074304 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.074285 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rj8\" (UniqueName: \"kubernetes.io/projected/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-kube-api-access-p7rj8\") pod \"limitador-limitador-8588749977-r4z9t\" (UID: \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\") " pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:25.134376 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.134346 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r4z9t"] Sep 10 17:23:25.175509 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.175471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-config-file\") pod \"limitador-limitador-8588749977-r4z9t\" (UID: \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\") " pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:25.175700 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.175526 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rj8\" (UniqueName: \"kubernetes.io/projected/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-kube-api-access-p7rj8\") pod \"limitador-limitador-8588749977-r4z9t\" (UID: \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\") " pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:25.176121 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.176102 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-config-file\") pod \"limitador-limitador-8588749977-r4z9t\" (UID: \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\") " pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:25.201976 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.201948 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rj8\" (UniqueName: \"kubernetes.io/projected/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-kube-api-access-p7rj8\") pod \"limitador-limitador-8588749977-r4z9t\" (UID: \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\") " pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:25.353578 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.353497 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:25.654324 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.654296 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r4z9t"] Sep 10 17:23:25.655614 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:23:25.655585 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad010c8_ec06_4db5_853c_6fbbec8fd16a.slice/crio-96e953e3e931f7ab04930f7a61838ac32a4e597f33aa466e5f79bceadaa2a2d0 WatchSource:0}: Error finding container 96e953e3e931f7ab04930f7a61838ac32a4e597f33aa466e5f79bceadaa2a2d0: Status 404 returned error can't find the container with id 96e953e3e931f7ab04930f7a61838ac32a4e597f33aa466e5f79bceadaa2a2d0 Sep 10 17:23:25.830145 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.830112 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" event={"ID":"5ad010c8-ec06-4db5-853c-6fbbec8fd16a","Type":"ContainerStarted","Data":"96e953e3e931f7ab04930f7a61838ac32a4e597f33aa466e5f79bceadaa2a2d0"} Sep 10 17:23:25.938903 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.938822 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b67m2"] Sep 10 17:23:25.941871 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.941853 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" Sep 10 17:23:25.946237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.946216 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dsh28\"" Sep 10 17:23:25.962312 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.962289 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b67m2"] Sep 10 17:23:25.981368 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:25.981345 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxlhf\" (UniqueName: \"kubernetes.io/projected/e3440b5e-1a15-4310-aeb0-9533d335e664-kube-api-access-sxlhf\") pod \"authorino-7bcfd8f9cd-b67m2\" (UID: \"e3440b5e-1a15-4310-aeb0-9533d335e664\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" Sep 10 17:23:26.081898 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:26.081861 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlhf\" (UniqueName: \"kubernetes.io/projected/e3440b5e-1a15-4310-aeb0-9533d335e664-kube-api-access-sxlhf\") pod \"authorino-7bcfd8f9cd-b67m2\" (UID: \"e3440b5e-1a15-4310-aeb0-9533d335e664\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" Sep 10 17:23:26.092007 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:26.091977 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlhf\" (UniqueName: \"kubernetes.io/projected/e3440b5e-1a15-4310-aeb0-9533d335e664-kube-api-access-sxlhf\") pod \"authorino-7bcfd8f9cd-b67m2\" (UID: \"e3440b5e-1a15-4310-aeb0-9533d335e664\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" Sep 10 17:23:26.251496 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:26.251466 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" Sep 10 17:23:26.532850 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:26.532547 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b67m2"] Sep 10 17:23:26.630441 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:23:26.630408 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3440b5e_1a15_4310_aeb0_9533d335e664.slice/crio-23a39e02e5a7c80e84abe7eb8da4ebb9f2d65b0e51bb6a7359c1f3b2f13cde39 WatchSource:0}: Error finding container 23a39e02e5a7c80e84abe7eb8da4ebb9f2d65b0e51bb6a7359c1f3b2f13cde39: Status 404 returned error can't find the container with id 23a39e02e5a7c80e84abe7eb8da4ebb9f2d65b0e51bb6a7359c1f3b2f13cde39 Sep 10 17:23:26.834911 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:26.834811 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" event={"ID":"e3440b5e-1a15-4310-aeb0-9533d335e664","Type":"ContainerStarted","Data":"23a39e02e5a7c80e84abe7eb8da4ebb9f2d65b0e51bb6a7359c1f3b2f13cde39"} Sep 10 17:23:26.836343 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:26.836315 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" event={"ID":"5ad010c8-ec06-4db5-853c-6fbbec8fd16a","Type":"ContainerStarted","Data":"eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd"} Sep 10 17:23:26.836481 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:26.836414 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:26.862767 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:26.862716 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" podStartSLOduration=0.858364268 podStartE2EDuration="1.862700555s" totalImagesPullingTime="1.004336287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:23:26.859685298 +0000 UTC m=+605.957583672" watchObservedRunningTime="2026-09-10 17:23:26.862700555 +0000 UTC m=+605.960598916" Sep 10 17:23:28.845963 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:28.845924 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" event={"ID":"e3440b5e-1a15-4310-aeb0-9533d335e664","Type":"ContainerStarted","Data":"ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3"} Sep 10 17:23:28.870720 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:28.870654 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" podStartSLOduration=2.262177511 podStartE2EDuration="3.8706385s" totalImagesPullingTime="1.608460989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:23:28.868193006 +0000 UTC m=+607.966091367" watchObservedRunningTime="2026-09-10 17:23:28.8706385 +0000 UTC m=+607.968536861" Sep 10 17:23:29.715329 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:29.715289 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b67m2"] Sep 10 17:23:30.854767 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:30.854721 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" podUID="e3440b5e-1a15-4310-aeb0-9533d335e664" containerName="authorino" containerID="cri-o://ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3" gracePeriod=30 Sep 10 17:23:31.105792 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.105732 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" Sep 10 17:23:31.226799 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.226745 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxlhf\" (UniqueName: \"kubernetes.io/projected/e3440b5e-1a15-4310-aeb0-9533d335e664-kube-api-access-sxlhf\") pod \"e3440b5e-1a15-4310-aeb0-9533d335e664\" (UID: \"e3440b5e-1a15-4310-aeb0-9533d335e664\") " Sep 10 17:23:31.228788 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.228761 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3440b5e-1a15-4310-aeb0-9533d335e664-kube-api-access-sxlhf" (OuterVolumeSpecName: "kube-api-access-sxlhf") pod "e3440b5e-1a15-4310-aeb0-9533d335e664" (UID: "e3440b5e-1a15-4310-aeb0-9533d335e664"). InnerVolumeSpecName "kube-api-access-sxlhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:23:31.328196 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.328163 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxlhf\" (UniqueName: \"kubernetes.io/projected/e3440b5e-1a15-4310-aeb0-9533d335e664-kube-api-access-sxlhf\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:23:31.859892 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.859853 2557 generic.go:356] "Generic (PLEG): container finished" podID="e3440b5e-1a15-4310-aeb0-9533d335e664" containerID="ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3" exitCode=0 Sep 10 17:23:31.860352 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.859914 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" Sep 10 17:23:31.860352 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.859932 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" event={"ID":"e3440b5e-1a15-4310-aeb0-9533d335e664","Type":"ContainerDied","Data":"ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3"} Sep 10 17:23:31.860352 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.859976 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b67m2" event={"ID":"e3440b5e-1a15-4310-aeb0-9533d335e664","Type":"ContainerDied","Data":"23a39e02e5a7c80e84abe7eb8da4ebb9f2d65b0e51bb6a7359c1f3b2f13cde39"} Sep 10 17:23:31.860352 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.859998 2557 scope.go:117] "RemoveContainer" containerID="ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3" Sep 10 17:23:31.867799 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.867782 2557 scope.go:117] "RemoveContainer" containerID="ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3" Sep 10 17:23:31.868102 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:23:31.868084 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3\": container with ID starting with ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3 not found: ID does not exist" containerID="ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3" Sep 10 17:23:31.868159 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.868112 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3"} err="failed to get container status \"ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3\": rpc error: code = NotFound desc = could not find container \"ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3\": container with ID starting with ab0197f648ec08c44bff90cdf62173bec2e89b4d1b64a58849f5fb98a4c704a3 not found: ID does not exist" Sep 10 17:23:31.876902 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.876880 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b67m2"] Sep 10 17:23:31.881507 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:31.881482 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b67m2"] Sep 10 17:23:33.386561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:33.386528 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3440b5e-1a15-4310-aeb0-9533d335e664" path="/var/lib/kubelet/pods/e3440b5e-1a15-4310-aeb0-9533d335e664/volumes" Sep 10 17:23:37.841905 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:37.841877 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:51.784706 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:51.784611 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r4z9t"] Sep 10 17:23:51.785155 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:51.784818 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" podUID="5ad010c8-ec06-4db5-853c-6fbbec8fd16a" containerName="limitador" containerID="cri-o://eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd" gracePeriod=30 Sep 10 17:23:52.321684 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.321661 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:52.400801 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.400717 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-config-file\") pod \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\" (UID: \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\") " Sep 10 17:23:52.400940 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.400921 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7rj8\" (UniqueName: \"kubernetes.io/projected/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-kube-api-access-p7rj8\") pod \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\" (UID: \"5ad010c8-ec06-4db5-853c-6fbbec8fd16a\") " Sep 10 17:23:52.401173 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.401148 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-config-file" (OuterVolumeSpecName: "config-file") pod "5ad010c8-ec06-4db5-853c-6fbbec8fd16a" (UID: "5ad010c8-ec06-4db5-853c-6fbbec8fd16a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:23:52.402886 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.402855 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-kube-api-access-p7rj8" (OuterVolumeSpecName: "kube-api-access-p7rj8") pod "5ad010c8-ec06-4db5-853c-6fbbec8fd16a" (UID: "5ad010c8-ec06-4db5-853c-6fbbec8fd16a"). InnerVolumeSpecName "kube-api-access-p7rj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:23:52.502370 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.502334 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7rj8\" (UniqueName: \"kubernetes.io/projected/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-kube-api-access-p7rj8\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:23:52.502370 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.502361 2557 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ad010c8-ec06-4db5-853c-6fbbec8fd16a-config-file\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:23:52.948192 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.948158 2557 generic.go:356] "Generic (PLEG): container finished" podID="5ad010c8-ec06-4db5-853c-6fbbec8fd16a" containerID="eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd" exitCode=0 Sep 10 17:23:52.948594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.948199 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" Sep 10 17:23:52.948594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.948238 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" event={"ID":"5ad010c8-ec06-4db5-853c-6fbbec8fd16a","Type":"ContainerDied","Data":"eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd"} Sep 10 17:23:52.948594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.948289 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r4z9t" event={"ID":"5ad010c8-ec06-4db5-853c-6fbbec8fd16a","Type":"ContainerDied","Data":"96e953e3e931f7ab04930f7a61838ac32a4e597f33aa466e5f79bceadaa2a2d0"} Sep 10 17:23:52.948594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.948314 2557 scope.go:117] "RemoveContainer" containerID="eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd" Sep 10 17:23:52.956534 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.956519 2557 scope.go:117] "RemoveContainer" containerID="eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd" Sep 10 17:23:52.956796 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:23:52.956780 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd\": container with ID starting with eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd not found: ID does not exist" containerID="eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd" Sep 10 17:23:52.956856 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.956802 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd"} err="failed to get container status \"eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd\": rpc error: code = NotFound desc = could not find container \"eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd\": container with ID starting with eda3b89df9512b3b03d21651d723ebd17242d62f54c4888a285eb0b52e0bc4dd not found: ID does not exist" Sep 10 17:23:52.968629 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.968603 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r4z9t"] Sep 10 17:23:52.972345 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:52.972323 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r4z9t"] Sep 10 17:23:53.385071 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:53.385023 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ad010c8-ec06-4db5-853c-6fbbec8fd16a" path="/var/lib/kubelet/pods/5ad010c8-ec06-4db5-853c-6fbbec8fd16a/volumes" Sep 10 17:23:56.842114 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.842080 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9"] Sep 10 17:23:56.842554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.842374 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ad010c8-ec06-4db5-853c-6fbbec8fd16a" containerName="limitador" Sep 10 17:23:56.842554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.842386 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ad010c8-ec06-4db5-853c-6fbbec8fd16a" containerName="limitador" Sep 10 17:23:56.842554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.842403 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3440b5e-1a15-4310-aeb0-9533d335e664" containerName="authorino" Sep 10 17:23:56.842554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.842408 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3440b5e-1a15-4310-aeb0-9533d335e664" containerName="authorino" Sep 10 17:23:56.842554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.842475 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ad010c8-ec06-4db5-853c-6fbbec8fd16a" containerName="limitador" Sep 10 17:23:56.842554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.842489 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3440b5e-1a15-4310-aeb0-9533d335e664" containerName="authorino" Sep 10 17:23:56.849230 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.849206 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:23:56.852147 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.852122 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 17:23:56.852261 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.852122 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dq9c5\"" Sep 10 17:23:56.852610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.852591 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 10 17:23:56.852691 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.852618 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 17:23:56.864985 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.864955 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9"] Sep 10 17:23:56.938154 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.938116 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/455b8f60-aa7a-465e-8177-51b794eddc94-data\") pod \"postgres-69fb4594cf-ws6w9\" (UID: \"455b8f60-aa7a-465e-8177-51b794eddc94\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:23:56.938154 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:56.938151 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl96k\" (UniqueName: \"kubernetes.io/projected/455b8f60-aa7a-465e-8177-51b794eddc94-kube-api-access-rl96k\") pod \"postgres-69fb4594cf-ws6w9\" (UID: \"455b8f60-aa7a-465e-8177-51b794eddc94\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:23:57.038911 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:57.038867 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/455b8f60-aa7a-465e-8177-51b794eddc94-data\") pod \"postgres-69fb4594cf-ws6w9\" (UID: \"455b8f60-aa7a-465e-8177-51b794eddc94\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:23:57.038911 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:57.038904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl96k\" (UniqueName: \"kubernetes.io/projected/455b8f60-aa7a-465e-8177-51b794eddc94-kube-api-access-rl96k\") pod \"postgres-69fb4594cf-ws6w9\" (UID: \"455b8f60-aa7a-465e-8177-51b794eddc94\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:23:57.039261 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:57.039241 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/455b8f60-aa7a-465e-8177-51b794eddc94-data\") pod \"postgres-69fb4594cf-ws6w9\" (UID: \"455b8f60-aa7a-465e-8177-51b794eddc94\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:23:57.053015 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:57.052985 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl96k\" (UniqueName: \"kubernetes.io/projected/455b8f60-aa7a-465e-8177-51b794eddc94-kube-api-access-rl96k\") pod \"postgres-69fb4594cf-ws6w9\" (UID: \"455b8f60-aa7a-465e-8177-51b794eddc94\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:23:57.158533 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:57.158448 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:23:57.441230 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:57.441202 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9"] Sep 10 17:23:57.443165 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:23:57.443138 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455b8f60_aa7a_465e_8177_51b794eddc94.slice/crio-8a5358e4a7c42b993419257c88f883b357da6624d702700aa9e4b76de238712c WatchSource:0}: Error finding container 8a5358e4a7c42b993419257c88f883b357da6624d702700aa9e4b76de238712c: Status 404 returned error can't find the container with id 8a5358e4a7c42b993419257c88f883b357da6624d702700aa9e4b76de238712c Sep 10 17:23:57.977016 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:23:57.976979 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" event={"ID":"455b8f60-aa7a-465e-8177-51b794eddc94","Type":"ContainerStarted","Data":"8a5358e4a7c42b993419257c88f883b357da6624d702700aa9e4b76de238712c"} Sep 10 17:24:03.001908 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:03.001872 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" event={"ID":"455b8f60-aa7a-465e-8177-51b794eddc94","Type":"ContainerStarted","Data":"8061a10e6ca5c7b80f46ac3d0fa0f749a7c239edf4f72d736c81d89cc26de7bd"} Sep 10 17:24:03.002317 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:03.001940 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:24:03.045002 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:03.044952 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" podStartSLOduration=2.2263245720000002 podStartE2EDuration="7.044938138s" totalImagesPullingTime="4.818613566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:03.044447384 +0000 UTC m=+642.142345758" watchObservedRunningTime="2026-09-10 17:24:03.044938138 +0000 UTC m=+642.142836499" Sep 10 17:24:09.033560 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:09.033531 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ws6w9" Sep 10 17:24:09.615376 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:09.615343 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-jbwvv"] Sep 10 17:24:09.618529 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:09.618513 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jbwvv" Sep 10 17:24:09.621114 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:09.621087 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dsh28\"" Sep 10 17:24:09.633613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:09.633590 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-jbwvv"] Sep 10 17:24:09.686393 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:09.686368 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-jbwvv"] Sep 10 17:24:09.686576 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:24:09.686560 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-g7kv2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-jbwvv" podUID="de7224e2-808d-4219-b26c-5042e0bd160f" Sep 10 17:24:09.739559 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:09.739534 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7kv2\" (UniqueName: \"kubernetes.io/projected/de7224e2-808d-4219-b26c-5042e0bd160f-kube-api-access-g7kv2\") pod \"authorino-557f876f69-jbwvv\" (UID: \"de7224e2-808d-4219-b26c-5042e0bd160f\") " pod="kuadrant-system/authorino-557f876f69-jbwvv" Sep 10 17:24:09.840835 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:09.840808 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7kv2\" (UniqueName: \"kubernetes.io/projected/de7224e2-808d-4219-b26c-5042e0bd160f-kube-api-access-g7kv2\") pod \"authorino-557f876f69-jbwvv\" (UID: \"de7224e2-808d-4219-b26c-5042e0bd160f\") " pod="kuadrant-system/authorino-557f876f69-jbwvv" Sep 10 17:24:09.851155 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:09.851131 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7kv2\" (UniqueName: \"kubernetes.io/projected/de7224e2-808d-4219-b26c-5042e0bd160f-kube-api-access-g7kv2\") pod \"authorino-557f876f69-jbwvv\" (UID: \"de7224e2-808d-4219-b26c-5042e0bd160f\") " pod="kuadrant-system/authorino-557f876f69-jbwvv" Sep 10 17:24:10.031146 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:10.031115 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jbwvv" Sep 10 17:24:10.037802 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:10.037782 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jbwvv" Sep 10 17:24:10.142602 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:10.142578 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7kv2\" (UniqueName: \"kubernetes.io/projected/de7224e2-808d-4219-b26c-5042e0bd160f-kube-api-access-g7kv2\") pod \"de7224e2-808d-4219-b26c-5042e0bd160f\" (UID: \"de7224e2-808d-4219-b26c-5042e0bd160f\") " Sep 10 17:24:10.144582 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:10.144558 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7224e2-808d-4219-b26c-5042e0bd160f-kube-api-access-g7kv2" (OuterVolumeSpecName: "kube-api-access-g7kv2") pod "de7224e2-808d-4219-b26c-5042e0bd160f" (UID: "de7224e2-808d-4219-b26c-5042e0bd160f"). InnerVolumeSpecName "kube-api-access-g7kv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:24:10.243890 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:10.243869 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7kv2\" (UniqueName: \"kubernetes.io/projected/de7224e2-808d-4219-b26c-5042e0bd160f-kube-api-access-g7kv2\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:24:11.035689 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:11.035659 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-jbwvv" Sep 10 17:24:11.076965 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:11.076938 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-jbwvv"] Sep 10 17:24:11.079523 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:11.079499 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-jbwvv"] Sep 10 17:24:11.385613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:11.385537 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de7224e2-808d-4219-b26c-5042e0bd160f" path="/var/lib/kubelet/pods/de7224e2-808d-4219-b26c-5042e0bd160f/volumes" Sep 10 17:24:16.969718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.969685 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56769c848c-cqntz"] Sep 10 17:24:16.971694 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.971676 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:16.974636 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.974612 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 17:24:16.974761 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.974614 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 17:24:16.975001 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.974984 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 17:24:16.978573 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.978555 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8xsv\"" Sep 10 17:24:16.987837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.987808 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56769c848c-cqntz"] Sep 10 17:24:16.995421 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.995393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-metrics-certs\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:16.995561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.995433 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jgmt\" (UniqueName: \"kubernetes.io/projected/60cd6122-5444-4710-8295-8907b291b511-kube-api-access-9jgmt\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:16.995561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:16.995460 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-webhook-cert\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:17.096329 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.096289 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-metrics-certs\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:17.096329 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.096332 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jgmt\" (UniqueName: \"kubernetes.io/projected/60cd6122-5444-4710-8295-8907b291b511-kube-api-access-9jgmt\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:17.096532 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.096357 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-webhook-cert\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:17.096577 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:24:17.096543 2557 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 10 17:24:17.096658 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:24:17.096619 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-metrics-certs podName:60cd6122-5444-4710-8295-8907b291b511 nodeName:}" failed. No retries permitted until 2026-09-10 17:24:17.596602652 +0000 UTC m=+656.694500990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-metrics-certs") pod "maas-controller-56769c848c-cqntz" (UID: "60cd6122-5444-4710-8295-8907b291b511") : secret "maas-controller-metrics-tls" not found Sep 10 17:24:17.098765 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.098737 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-webhook-cert\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:17.116442 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.116413 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jgmt\" (UniqueName: \"kubernetes.io/projected/60cd6122-5444-4710-8295-8907b291b511-kube-api-access-9jgmt\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:17.193022 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.192985 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-4j5qz"] Sep 10 17:24:17.195041 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.195015 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.217693 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.217657 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-4j5qz"] Sep 10 17:24:17.298817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.298775 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-webhook-cert\") pod \"maas-controller-65bb9fc5b4-4j5qz\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.298975 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.298920 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpkx\" (UniqueName: \"kubernetes.io/projected/ce96cb77-d52b-462c-82d3-721b3f05f659-kube-api-access-xdpkx\") pod \"maas-controller-65bb9fc5b4-4j5qz\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.299084 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.298982 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-metrics-certs\") pod \"maas-controller-65bb9fc5b4-4j5qz\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.399936 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.399895 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-webhook-cert\") pod \"maas-controller-65bb9fc5b4-4j5qz\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.400136 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.399959 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpkx\" (UniqueName: \"kubernetes.io/projected/ce96cb77-d52b-462c-82d3-721b3f05f659-kube-api-access-xdpkx\") pod \"maas-controller-65bb9fc5b4-4j5qz\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.400136 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.399988 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-metrics-certs\") pod \"maas-controller-65bb9fc5b4-4j5qz\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.402448 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.402421 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-metrics-certs\") pod \"maas-controller-65bb9fc5b4-4j5qz\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.402559 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.402469 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-webhook-cert\") pod \"maas-controller-65bb9fc5b4-4j5qz\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.409269 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.409246 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpkx\" (UniqueName: \"kubernetes.io/projected/ce96cb77-d52b-462c-82d3-721b3f05f659-kube-api-access-xdpkx\") pod \"maas-controller-65bb9fc5b4-4j5qz\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.505381 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.505325 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:17.601522 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.601484 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-metrics-certs\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:17.603976 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.603944 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-metrics-certs\") pod \"maas-controller-56769c848c-cqntz\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:17.779551 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.779519 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-4j5qz"] Sep 10 17:24:17.780949 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:24:17.780924 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce96cb77_d52b_462c_82d3_721b3f05f659.slice/crio-f5d8cb56bdbcdb22b576afdc5f7b13b1bbe670af161d15b1e30d2f65893aaae3 WatchSource:0}: Error finding container f5d8cb56bdbcdb22b576afdc5f7b13b1bbe670af161d15b1e30d2f65893aaae3: Status 404 returned error can't find the container with id f5d8cb56bdbcdb22b576afdc5f7b13b1bbe670af161d15b1e30d2f65893aaae3 Sep 10 17:24:17.881675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:17.881586 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:18.068457 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:18.068423 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" event={"ID":"ce96cb77-d52b-462c-82d3-721b3f05f659","Type":"ContainerStarted","Data":"f5d8cb56bdbcdb22b576afdc5f7b13b1bbe670af161d15b1e30d2f65893aaae3"} Sep 10 17:24:18.156432 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:18.156356 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56769c848c-cqntz"] Sep 10 17:24:18.160092 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:24:18.160062 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cd6122_5444_4710_8295_8907b291b511.slice/crio-bb0835e96684066be011d7a9be4130955d7363c13985fdead753d0b1779e3e2b WatchSource:0}: Error finding container bb0835e96684066be011d7a9be4130955d7363c13985fdead753d0b1779e3e2b: Status 404 returned error can't find the container with id bb0835e96684066be011d7a9be4130955d7363c13985fdead753d0b1779e3e2b Sep 10 17:24:19.077844 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:19.077805 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56769c848c-cqntz" event={"ID":"60cd6122-5444-4710-8295-8907b291b511","Type":"ContainerStarted","Data":"bb0835e96684066be011d7a9be4130955d7363c13985fdead753d0b1779e3e2b"} Sep 10 17:24:21.088383 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:21.088344 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56769c848c-cqntz" event={"ID":"60cd6122-5444-4710-8295-8907b291b511","Type":"ContainerStarted","Data":"09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3"} Sep 10 17:24:21.088845 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:21.088406 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:21.089856 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:21.089828 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" event={"ID":"ce96cb77-d52b-462c-82d3-721b3f05f659","Type":"ContainerStarted","Data":"e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2"} Sep 10 17:24:21.089957 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:21.089881 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:21.116849 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:21.116806 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56769c848c-cqntz" podStartSLOduration=2.951531038 podStartE2EDuration="5.116791243s" totalImagesPullingTime="2.165260205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:21.116500556 +0000 UTC m=+660.214398919" watchObservedRunningTime="2026-09-10 17:24:21.116791243 +0000 UTC m=+660.214689603" Sep 10 17:24:21.139839 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:21.139776 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" podStartSLOduration=1.598381893 podStartE2EDuration="4.139754636s" totalImagesPullingTime="2.541372743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:21.139351515 +0000 UTC m=+660.237249876" watchObservedRunningTime="2026-09-10 17:24:21.139754636 +0000 UTC m=+660.237653001" Sep 10 17:24:32.097542 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.097505 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:32.097986 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.097587 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:24:32.221113 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.221081 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56769c848c-cqntz"] Sep 10 17:24:32.221337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.221309 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56769c848c-cqntz" podUID="60cd6122-5444-4710-8295-8907b291b511" containerName="manager" containerID="cri-o://09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3" gracePeriod=30 Sep 10 17:24:32.467322 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.467294 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:32.527141 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.527102 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jgmt\" (UniqueName: \"kubernetes.io/projected/60cd6122-5444-4710-8295-8907b291b511-kube-api-access-9jgmt\") pod \"60cd6122-5444-4710-8295-8907b291b511\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " Sep 10 17:24:32.527141 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.527143 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-webhook-cert\") pod \"60cd6122-5444-4710-8295-8907b291b511\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " Sep 10 17:24:32.527399 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.527180 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-metrics-certs\") pod \"60cd6122-5444-4710-8295-8907b291b511\" (UID: \"60cd6122-5444-4710-8295-8907b291b511\") " Sep 10 17:24:32.529596 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.529559 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "60cd6122-5444-4710-8295-8907b291b511" (UID: "60cd6122-5444-4710-8295-8907b291b511"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:24:32.529728 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.529570 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60cd6122-5444-4710-8295-8907b291b511-kube-api-access-9jgmt" (OuterVolumeSpecName: "kube-api-access-9jgmt") pod "60cd6122-5444-4710-8295-8907b291b511" (UID: "60cd6122-5444-4710-8295-8907b291b511"). InnerVolumeSpecName "kube-api-access-9jgmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:24:32.529728 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.529668 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "60cd6122-5444-4710-8295-8907b291b511" (UID: "60cd6122-5444-4710-8295-8907b291b511"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:24:32.628000 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.627961 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jgmt\" (UniqueName: \"kubernetes.io/projected/60cd6122-5444-4710-8295-8907b291b511-kube-api-access-9jgmt\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:24:32.628000 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.627995 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-webhook-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:24:32.628000 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:32.628005 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60cd6122-5444-4710-8295-8907b291b511-metrics-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:24:33.142824 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.142786 2557 generic.go:356] "Generic (PLEG): container finished" podID="60cd6122-5444-4710-8295-8907b291b511" containerID="09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3" exitCode=0 Sep 10 17:24:33.143247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.142829 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56769c848c-cqntz" Sep 10 17:24:33.143247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.142875 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56769c848c-cqntz" event={"ID":"60cd6122-5444-4710-8295-8907b291b511","Type":"ContainerDied","Data":"09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3"} Sep 10 17:24:33.143247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.142935 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56769c848c-cqntz" event={"ID":"60cd6122-5444-4710-8295-8907b291b511","Type":"ContainerDied","Data":"bb0835e96684066be011d7a9be4130955d7363c13985fdead753d0b1779e3e2b"} Sep 10 17:24:33.143247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.142958 2557 scope.go:117] "RemoveContainer" containerID="09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3" Sep 10 17:24:33.152071 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.152020 2557 scope.go:117] "RemoveContainer" containerID="09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3" Sep 10 17:24:33.152338 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:24:33.152323 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3\": container with ID starting with 09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3 not found: ID does not exist" containerID="09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3" Sep 10 17:24:33.152392 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.152347 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3"} err="failed to get container status \"09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3\": rpc error: code = NotFound desc = could not find container \"09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3\": container with ID starting with 09af7795fd927028e5303b587b0dcdb563afc788a3911a8bf398806002c504b3 not found: ID does not exist" Sep 10 17:24:33.176540 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.176507 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56769c848c-cqntz"] Sep 10 17:24:33.184705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.184675 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56769c848c-cqntz"] Sep 10 17:24:33.387096 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:33.387062 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60cd6122-5444-4710-8295-8907b291b511" path="/var/lib/kubelet/pods/60cd6122-5444-4710-8295-8907b291b511/volumes" Sep 10 17:24:54.274183 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.274146 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb"] Sep 10 17:24:54.274634 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.274522 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60cd6122-5444-4710-8295-8907b291b511" containerName="manager" Sep 10 17:24:54.274634 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.274537 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="60cd6122-5444-4710-8295-8907b291b511" containerName="manager" Sep 10 17:24:54.274634 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.274596 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="60cd6122-5444-4710-8295-8907b291b511" containerName="manager" Sep 10 17:24:54.278460 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.278444 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.299550 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.299524 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 17:24:54.299815 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.299798 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 17:24:54.300062 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.300029 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2hfkp\"" Sep 10 17:24:54.316856 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.316827 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb"] Sep 10 17:24:54.417687 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.417648 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nds5h\" (UniqueName: \"kubernetes.io/projected/e980a408-f795-463a-852d-5386cc740a1c-kube-api-access-nds5h\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.417874 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.417700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e980a408-f795-463a-852d-5386cc740a1c-openshift-service-ca\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.417874 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.417721 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-metrics-certs\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.417874 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.417808 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-maas-api-tls\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.518859 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.518790 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nds5h\" (UniqueName: \"kubernetes.io/projected/e980a408-f795-463a-852d-5386cc740a1c-kube-api-access-nds5h\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.519083 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.518892 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e980a408-f795-463a-852d-5386cc740a1c-openshift-service-ca\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.519083 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.518926 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-metrics-certs\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.519083 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.518964 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-maas-api-tls\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.519281 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:24:54.519084 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 10 17:24:54.519281 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:24:54.519178 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-metrics-certs podName:e980a408-f795-463a-852d-5386cc740a1c nodeName:}" failed. No retries permitted until 2026-09-10 17:24:55.019153843 +0000 UTC m=+694.117052183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-metrics-certs") pod "maas-api-6c68568df5-xrkdb" (UID: "e980a408-f795-463a-852d-5386cc740a1c") : secret "maas-api-metrics-tls" not found Sep 10 17:24:54.519564 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.519543 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e980a408-f795-463a-852d-5386cc740a1c-openshift-service-ca\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.521528 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.521506 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-maas-api-tls\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:54.558914 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:54.558843 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nds5h\" (UniqueName: \"kubernetes.io/projected/e980a408-f795-463a-852d-5386cc740a1c-kube-api-access-nds5h\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:55.024402 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.024365 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-metrics-certs\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:55.026928 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.026898 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-metrics-certs\") pod \"maas-api-6c68568df5-xrkdb\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:55.187408 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.187371 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:24:55.385843 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.385750 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j"] Sep 10 17:24:55.388240 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.388218 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:24:55.391209 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.391187 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 17:24:55.428113 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.428076 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j"] Sep 10 17:24:55.486346 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.486310 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb"] Sep 10 17:24:55.491190 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:24:55.491161 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode980a408_f795_463a_852d_5386cc740a1c.slice/crio-72d41c68ac133adfc4298605211275b0869836608bd2cd37d2e01ca6126c2f90 WatchSource:0}: Error finding container 72d41c68ac133adfc4298605211275b0869836608bd2cd37d2e01ca6126c2f90: Status 404 returned error can't find the container with id 72d41c68ac133adfc4298605211275b0869836608bd2cd37d2e01ca6126c2f90 Sep 10 17:24:55.492514 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.492498 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:24:55.514860 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.514829 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w7ck4"] Sep 10 17:24:55.516935 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.516920 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:24:55.529553 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.529520 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b251030e-8917-4c97-a641-542c0dc76045-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lp52j\" (UID: \"b251030e-8917-4c97-a641-542c0dc76045\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:24:55.529717 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.529600 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p97z\" (UniqueName: \"kubernetes.io/projected/b251030e-8917-4c97-a641-542c0dc76045-kube-api-access-2p97z\") pod \"payload-pre-processing-59fbc47bf8-lp52j\" (UID: \"b251030e-8917-4c97-a641-542c0dc76045\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:24:55.532373 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.532352 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-z85rz\"" Sep 10 17:24:55.555517 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.555485 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w7ck4"] Sep 10 17:24:55.631605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.631557 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpdsk\" (UniqueName: \"kubernetes.io/projected/c21f986f-64b2-4347-af53-2d38959e718f-kube-api-access-mpdsk\") pod \"payload-processing-5b54b7fbb7-w7ck4\" (UID: \"c21f986f-64b2-4347-af53-2d38959e718f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:24:55.631809 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.631623 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p97z\" (UniqueName: \"kubernetes.io/projected/b251030e-8917-4c97-a641-542c0dc76045-kube-api-access-2p97z\") pod \"payload-pre-processing-59fbc47bf8-lp52j\" (UID: \"b251030e-8917-4c97-a641-542c0dc76045\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:24:55.631809 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.631706 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21f986f-64b2-4347-af53-2d38959e718f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-w7ck4\" (UID: \"c21f986f-64b2-4347-af53-2d38959e718f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:24:55.631809 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.631802 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b251030e-8917-4c97-a641-542c0dc76045-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lp52j\" (UID: \"b251030e-8917-4c97-a641-542c0dc76045\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:24:55.632501 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.632481 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b251030e-8917-4c97-a641-542c0dc76045-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lp52j\" (UID: \"b251030e-8917-4c97-a641-542c0dc76045\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:24:55.650436 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.650372 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p97z\" (UniqueName: \"kubernetes.io/projected/b251030e-8917-4c97-a641-542c0dc76045-kube-api-access-2p97z\") pod \"payload-pre-processing-59fbc47bf8-lp52j\" (UID: \"b251030e-8917-4c97-a641-542c0dc76045\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:24:55.698527 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.698489 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:24:55.732928 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.732896 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpdsk\" (UniqueName: \"kubernetes.io/projected/c21f986f-64b2-4347-af53-2d38959e718f-kube-api-access-mpdsk\") pod \"payload-processing-5b54b7fbb7-w7ck4\" (UID: \"c21f986f-64b2-4347-af53-2d38959e718f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:24:55.733130 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.733004 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21f986f-64b2-4347-af53-2d38959e718f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-w7ck4\" (UID: \"c21f986f-64b2-4347-af53-2d38959e718f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:24:55.733841 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.733817 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21f986f-64b2-4347-af53-2d38959e718f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-w7ck4\" (UID: \"c21f986f-64b2-4347-af53-2d38959e718f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:24:55.744883 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.744853 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpdsk\" (UniqueName: \"kubernetes.io/projected/c21f986f-64b2-4347-af53-2d38959e718f-kube-api-access-mpdsk\") pod \"payload-processing-5b54b7fbb7-w7ck4\" (UID: \"c21f986f-64b2-4347-af53-2d38959e718f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:24:55.834841 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:55.834794 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:24:56.014214 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:56.014170 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j"] Sep 10 17:24:56.017214 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:24:56.017189 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb251030e_8917_4c97_a641_542c0dc76045.slice/crio-fe49b5d0e17f1ec0d085535414e7f0c7a7896ff8e71625c3dd68efbb7f1a5066 WatchSource:0}: Error finding container fe49b5d0e17f1ec0d085535414e7f0c7a7896ff8e71625c3dd68efbb7f1a5066: Status 404 returned error can't find the container with id fe49b5d0e17f1ec0d085535414e7f0c7a7896ff8e71625c3dd68efbb7f1a5066 Sep 10 17:24:56.141858 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:56.141830 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w7ck4"] Sep 10 17:24:56.142501 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:24:56.142472 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21f986f_64b2_4347_af53_2d38959e718f.slice/crio-2fd3f4345c9e2ead2addcc7af7ec6f12205ee3886df4f94980d1039c8a6fc51d WatchSource:0}: Error finding container 2fd3f4345c9e2ead2addcc7af7ec6f12205ee3886df4f94980d1039c8a6fc51d: Status 404 returned error can't find the container with id 2fd3f4345c9e2ead2addcc7af7ec6f12205ee3886df4f94980d1039c8a6fc51d Sep 10 17:24:56.262057 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:56.260329 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" event={"ID":"b251030e-8917-4c97-a641-542c0dc76045","Type":"ContainerStarted","Data":"fe49b5d0e17f1ec0d085535414e7f0c7a7896ff8e71625c3dd68efbb7f1a5066"} Sep 10 17:24:56.270431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:56.270389 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" event={"ID":"e980a408-f795-463a-852d-5386cc740a1c","Type":"ContainerStarted","Data":"72d41c68ac133adfc4298605211275b0869836608bd2cd37d2e01ca6126c2f90"} Sep 10 17:24:56.272003 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:24:56.271963 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" event={"ID":"c21f986f-64b2-4347-af53-2d38959e718f","Type":"ContainerStarted","Data":"2fd3f4345c9e2ead2addcc7af7ec6f12205ee3886df4f94980d1039c8a6fc51d"} Sep 10 17:25:00.297302 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:00.297270 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" event={"ID":"b251030e-8917-4c97-a641-542c0dc76045","Type":"ContainerStarted","Data":"1543a6ffcfae9b0da416e7873992c5ac84bc6d33867502e23a398956b4623488"} Sep 10 17:25:00.297775 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:00.297374 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:25:00.298620 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:00.298589 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" event={"ID":"e980a408-f795-463a-852d-5386cc740a1c","Type":"ContainerStarted","Data":"33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1"} Sep 10 17:25:00.298729 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:00.298693 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:25:00.300181 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:00.300163 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" event={"ID":"c21f986f-64b2-4347-af53-2d38959e718f","Type":"ContainerStarted","Data":"9da34318303ac98ac3f5f8d8d3730d659ce59385668b471b0ff0f315c363f983"} Sep 10 17:25:00.300275 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:00.300201 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:25:00.317045 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:00.316990 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" podStartSLOduration=1.862707818 podStartE2EDuration="5.316973014s" totalImagesPullingTime="3.454265196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:00.316450416 +0000 UTC m=+699.414348777" watchObservedRunningTime="2026-09-10 17:25:00.316973014 +0000 UTC m=+699.414871377" Sep 10 17:25:00.334926 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:00.334869 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" podStartSLOduration=2.050942481 podStartE2EDuration="5.33485486s" totalImagesPullingTime="3.283912379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:00.333778551 +0000 UTC m=+699.431676913" watchObservedRunningTime="2026-09-10 17:25:00.33485486 +0000 UTC m=+699.432753223" Sep 10 17:25:05.325579 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:05.325541 2557 generic.go:356] "Generic (PLEG): container finished" podID="0578c7d9-e911-4d2d-9266-7862658d07f4" containerID="18727ab7b87bd7d7b315fb1928a8960ab5d52fa0aa0b99bf1c742abe2560a466" exitCode=0 Sep 10 17:25:05.326018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:05.325616 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" event={"ID":"0578c7d9-e911-4d2d-9266-7862658d07f4","Type":"ContainerDied","Data":"18727ab7b87bd7d7b315fb1928a8960ab5d52fa0aa0b99bf1c742abe2560a466"} Sep 10 17:25:05.326018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:05.326000 2557 scope.go:117] "RemoveContainer" containerID="18727ab7b87bd7d7b315fb1928a8960ab5d52fa0aa0b99bf1c742abe2560a466" Sep 10 17:25:05.327029 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:05.326959 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:25:05.327029 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:05.327027 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:25:05.327178 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:05.327073 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:25:05.349779 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:05.349736 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" podStartSLOduration=7.425809008 podStartE2EDuration="11.349719739s" totalImagesPullingTime="3.923910731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:00.358918982 +0000 UTC m=+699.456817345" watchObservedRunningTime="2026-09-10 17:25:05.349719739 +0000 UTC m=+704.447618099" Sep 10 17:25:06.311591 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:06.311560 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:25:06.331365 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:06.331331 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" event={"ID":"0578c7d9-e911-4d2d-9266-7862658d07f4","Type":"ContainerStarted","Data":"fca78875943dcdc854392c9ecf2154d1243cee59d8bac8e7851669e2c200d2fb"} Sep 10 17:25:06.808656 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:06.808618 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:25:06.809752 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:06.809723 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8k8f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Sep 10 17:25:06.809884 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:06.809786 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" podUID="0578c7d9-e911-4d2d-9266-7862658d07f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Sep 10 17:25:07.808792 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:07.808747 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8k8f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Sep 10 17:25:07.809221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:07.808820 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" podUID="0578c7d9-e911-4d2d-9266-7862658d07f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Sep 10 17:25:08.808909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:08.808878 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8k8f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Sep 10 17:25:08.809299 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:08.808931 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" podUID="0578c7d9-e911-4d2d-9266-7862658d07f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Sep 10 17:25:09.808403 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:09.808367 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8k8f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Sep 10 17:25:09.808575 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:09.808436 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" podUID="0578c7d9-e911-4d2d-9266-7862658d07f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Sep 10 17:25:09.825502 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:09.825471 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:25:10.808718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:10.808682 2557 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-8k8f9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body= Sep 10 17:25:10.808900 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:10.808741 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" podUID="0578c7d9-e911-4d2d-9266-7862658d07f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" Sep 10 17:25:11.308029 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:11.307994 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lp52j" Sep 10 17:25:11.308497 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:11.308109 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-w7ck4" Sep 10 17:25:11.812261 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:11.812234 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:25:11.813150 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:11.813129 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-8k8f9" Sep 10 17:25:13.855926 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.855839 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f"] Sep 10 17:25:13.858221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.858198 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:13.862522 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.862494 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 17:25:13.862978 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.862949 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 17:25:13.863113 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.863022 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 17:25:13.863113 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.863026 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sb45l\"" Sep 10 17:25:13.880657 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.880634 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f"] Sep 10 17:25:13.917104 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.917072 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58kw2\" (UniqueName: \"kubernetes.io/projected/f2b73d20-8756-487d-a6ee-8892bc1fa561-kube-api-access-58kw2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:13.917214 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.917112 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:13.917214 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.917203 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:13.917321 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.917229 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:13.917321 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.917254 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b73d20-8756-487d-a6ee-8892bc1fa561-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:13.917401 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:13.917334 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.018608 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.018575 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.018608 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.018609 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.018790 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.018625 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b73d20-8756-487d-a6ee-8892bc1fa561-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.018790 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.018650 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.018890 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.018861 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58kw2\" (UniqueName: \"kubernetes.io/projected/f2b73d20-8756-487d-a6ee-8892bc1fa561-kube-api-access-58kw2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.018943 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.018910 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.018992 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.018971 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.019068 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.019020 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.019292 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.019265 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.020808 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.020792 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2b73d20-8756-487d-a6ee-8892bc1fa561-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.021189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.021171 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2b73d20-8756-487d-a6ee-8892bc1fa561-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.032942 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.032921 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58kw2\" (UniqueName: \"kubernetes.io/projected/f2b73d20-8756-487d-a6ee-8892bc1fa561-kube-api-access-58kw2\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f\" (UID: \"f2b73d20-8756-487d-a6ee-8892bc1fa561\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.168306 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.168241 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:14.486884 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:14.486791 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f"] Sep 10 17:25:14.490208 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:25:14.490176 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b73d20_8756_487d_a6ee_8892bc1fa561.slice/crio-3771ceb1232435de7743ad276d23180b39fb3311fd0592d099835be07d3c1265 WatchSource:0}: Error finding container 3771ceb1232435de7743ad276d23180b39fb3311fd0592d099835be07d3c1265: Status 404 returned error can't find the container with id 3771ceb1232435de7743ad276d23180b39fb3311fd0592d099835be07d3c1265 Sep 10 17:25:15.378959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:15.378919 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" event={"ID":"f2b73d20-8756-487d-a6ee-8892bc1fa561","Type":"ContainerStarted","Data":"3771ceb1232435de7743ad276d23180b39fb3311fd0592d099835be07d3c1265"} Sep 10 17:25:16.386075 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:16.386018 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" event={"ID":"f2b73d20-8756-487d-a6ee-8892bc1fa561","Type":"ContainerStarted","Data":"9b2f1a137aa9e77176c96aa4c0f8403632e8c5b669512a70f487985bb2799347"} Sep 10 17:25:16.408925 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:16.408860 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" podStartSLOduration=2.279686402 podStartE2EDuration="3.408838676s" totalImagesPullingTime="1.129152274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:16.406819534 +0000 UTC m=+715.504717907" watchObservedRunningTime="2026-09-10 17:25:16.408838676 +0000 UTC m=+715.506737058" Sep 10 17:25:24.169050 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:24.169003 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:24.169423 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:24.169086 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:24.181259 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:24.181227 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:24.430950 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:24.430870 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f" Sep 10 17:25:34.968360 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:34.968326 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws"] Sep 10 17:25:34.976052 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:34.976013 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:34.981049 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:34.981012 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 17:25:35.001636 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.001605 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws"] Sep 10 17:25:35.107578 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.107546 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4864\" (UniqueName: \"kubernetes.io/projected/8da6b695-6834-4768-87f2-d5a29a900678-kube-api-access-h4864\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.107744 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.107624 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.107744 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.107650 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.107744 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.107675 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da6b695-6834-4768-87f2-d5a29a900678-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.107870 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.107778 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.107870 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.107838 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.208227 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.208190 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da6b695-6834-4768-87f2-d5a29a900678-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.208393 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.208237 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.208393 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.208279 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.208393 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.208324 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4864\" (UniqueName: \"kubernetes.io/projected/8da6b695-6834-4768-87f2-d5a29a900678-kube-api-access-h4864\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.208542 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.208392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.208542 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.208425 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.208731 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.208703 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.208835 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.208781 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.208876 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.208853 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.210615 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.210597 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8da6b695-6834-4768-87f2-d5a29a900678-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.210819 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.210803 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8da6b695-6834-4768-87f2-d5a29a900678-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.223515 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.223455 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4864\" (UniqueName: \"kubernetes.io/projected/8da6b695-6834-4768-87f2-d5a29a900678-kube-api-access-h4864\") pod \"e2e-distinct-simulated-kserve-59975d69f9-t86ws\" (UID: \"8da6b695-6834-4768-87f2-d5a29a900678\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.286310 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.286274 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:35.568894 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:35.568864 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws"] Sep 10 17:25:35.571772 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:25:35.571738 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da6b695_6834_4768_87f2_d5a29a900678.slice/crio-937741956a01a79f2f12d260e37acdf916cb8eb28cf87ce6afaeda34f5435c1b WatchSource:0}: Error finding container 937741956a01a79f2f12d260e37acdf916cb8eb28cf87ce6afaeda34f5435c1b: Status 404 returned error can't find the container with id 937741956a01a79f2f12d260e37acdf916cb8eb28cf87ce6afaeda34f5435c1b Sep 10 17:25:36.473004 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:36.472966 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" event={"ID":"8da6b695-6834-4768-87f2-d5a29a900678","Type":"ContainerStarted","Data":"2a4bc86aace6e3d270454d67a258a054c2d06dadd200b7496176cc7478e6c10e"} Sep 10 17:25:36.473441 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:36.473019 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" event={"ID":"8da6b695-6834-4768-87f2-d5a29a900678","Type":"ContainerStarted","Data":"937741956a01a79f2f12d260e37acdf916cb8eb28cf87ce6afaeda34f5435c1b"} Sep 10 17:25:36.505881 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:36.505832 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" podStartSLOduration=2.269891502 podStartE2EDuration="2.505816238s" totalImagesPullingTime="235.924736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:36.504269556 +0000 UTC m=+735.602167918" watchObservedRunningTime="2026-09-10 17:25:36.505816238 +0000 UTC m=+735.603714598" Sep 10 17:25:45.286425 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:45.286394 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:45.286824 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:45.286450 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:45.299290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:45.299265 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:25:45.528105 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:25:45.528074 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-t86ws" Sep 10 17:27:44.497262 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.497225 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2"] Sep 10 17:27:44.500719 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.500694 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.503641 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.503622 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8c46e8-openshift-default-dockercfg-l7qwv\"" Sep 10 17:27:44.528775 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.528734 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2"] Sep 10 17:27:44.635746 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.635705 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7360973-3aae-4c11-956b-348bd684e3c0-istio-podinfo\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.635746 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.635745 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-socket\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.635944 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.635832 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-istio-token\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.635944 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.635911 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-envoy\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.635944 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.635933 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-certs\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.636111 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.635948 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7360973-3aae-4c11-956b-348bd684e3c0-istiod-ca-cert\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.636111 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.635966 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pskdn\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-kube-api-access-pskdn\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.636111 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.636008 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-credential-socket\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.636111 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.636024 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-data\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.736831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.736789 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-envoy\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.736839 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-certs\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.736954 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7360973-3aae-4c11-956b-348bd684e3c0-istiod-ca-cert\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.736990 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pskdn\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-kube-api-access-pskdn\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737229 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737057 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-credential-socket\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737229 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737083 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-data\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737229 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737170 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7360973-3aae-4c11-956b-348bd684e3c0-istio-podinfo\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737229 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737203 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-socket\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737428 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737248 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-istio-token\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737428 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737256 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-certs\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737428 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737363 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-credential-socket\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737568 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-data\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737605 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737597 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-socket\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.737748 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.737727 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7360973-3aae-4c11-956b-348bd684e3c0-istiod-ca-cert\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.739537 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.739506 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-envoy\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.739957 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.739938 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7360973-3aae-4c11-956b-348bd684e3c0-istio-podinfo\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.748489 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.748433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-istio-token\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.748600 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.748556 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pskdn\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-kube-api-access-pskdn\") pod \"e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:44.811667 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:44.811629 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:45.116674 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:45.116645 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2"] Sep 10 17:27:45.118106 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:27:45.118080 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7360973_3aae_4c11_956b_348bd684e3c0.slice/crio-2432a4aaa1c0b84c8667e354b38ae4154bbf88ec7ef9bed082b378a19f1047df WatchSource:0}: Error finding container 2432a4aaa1c0b84c8667e354b38ae4154bbf88ec7ef9bed082b378a19f1047df: Status 404 returned error can't find the container with id 2432a4aaa1c0b84c8667e354b38ae4154bbf88ec7ef9bed082b378a19f1047df Sep 10 17:27:45.120328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:45.120288 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:27:45.120399 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:45.120372 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:27:45.120442 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:45.120412 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:27:46.060283 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:46.060242 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" event={"ID":"a7360973-3aae-4c11-956b-348bd684e3c0","Type":"ContainerStarted","Data":"e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b"} Sep 10 17:27:46.060676 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:46.060333 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" event={"ID":"a7360973-3aae-4c11-956b-348bd684e3c0","Type":"ContainerStarted","Data":"2432a4aaa1c0b84c8667e354b38ae4154bbf88ec7ef9bed082b378a19f1047df"} Sep 10 17:27:46.108670 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:46.108613 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" podStartSLOduration=2.108597316 podStartE2EDuration="2.108597316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:46.103885839 +0000 UTC m=+865.201784202" watchObservedRunningTime="2026-09-10 17:27:46.108597316 +0000 UTC m=+865.206495677" Sep 10 17:27:46.812105 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:46.812066 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:46.816748 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:46.816723 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:47.065783 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:47.065690 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:47.066629 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:47.066605 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:27:49.614793 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.614761 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx"] Sep 10 17:27:49.618191 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.618167 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.620545 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.620523 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8c46e8-serving-cert\"" Sep 10 17:27:49.632907 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.632882 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx"] Sep 10 17:27:49.677292 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.677254 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caba8334-452f-4598-9110-764809181b0c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.677292 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.677294 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlj2\" (UniqueName: \"kubernetes.io/projected/caba8334-452f-4598-9110-764809181b0c-kube-api-access-6xlj2\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.677491 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.677366 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.677491 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.677392 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-metrics-certs\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.778423 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.778383 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caba8334-452f-4598-9110-764809181b0c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.778601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.778437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlj2\" (UniqueName: \"kubernetes.io/projected/caba8334-452f-4598-9110-764809181b0c-kube-api-access-6xlj2\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.778601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.778489 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.778601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.778512 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-metrics-certs\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.778794 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:27:49.778656 2557 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-serving-cert: secret "maas-api-e2e-inf-a-8c46e8-serving-cert" not found Sep 10 17:27:49.778794 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:27:49.778738 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-maas-api-tls podName:caba8334-452f-4598-9110-764809181b0c nodeName:}" failed. No retries permitted until 2026-09-10 17:27:50.27871558 +0000 UTC m=+869.376613942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-maas-api-tls") pod "maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" (UID: "caba8334-452f-4598-9110-764809181b0c") : secret "maas-api-e2e-inf-a-8c46e8-serving-cert" not found Sep 10 17:27:49.779241 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.779194 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caba8334-452f-4598-9110-764809181b0c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.780971 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.780940 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-metrics-certs\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:49.793330 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:49.793308 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlj2\" (UniqueName: \"kubernetes.io/projected/caba8334-452f-4598-9110-764809181b0c-kube-api-access-6xlj2\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:50.283236 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.283201 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:50.286252 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.286224 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:50.530648 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.530604 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:50.672145 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.672112 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w"] Sep 10 17:27:50.675714 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.675696 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:27:50.681306 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.681281 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8c46e8\"" Sep 10 17:27:50.681431 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.681327 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8c46e8-dockercfg-4nqx9\"" Sep 10 17:27:50.684187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.684158 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w"] Sep 10 17:27:50.686270 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.686244 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9334b0a0-9346-4db0-bdbc-1318669a6b1a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w\" (UID: \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:27:50.686376 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.686321 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtqr8\" (UniqueName: \"kubernetes.io/projected/9334b0a0-9346-4db0-bdbc-1318669a6b1a-kube-api-access-gtqr8\") pod \"payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w\" (UID: \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:27:50.786870 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.786785 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtqr8\" (UniqueName: \"kubernetes.io/projected/9334b0a0-9346-4db0-bdbc-1318669a6b1a-kube-api-access-gtqr8\") pod \"payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w\" (UID: \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:27:50.787077 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.786893 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9334b0a0-9346-4db0-bdbc-1318669a6b1a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w\" (UID: \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:27:50.787822 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.787798 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9334b0a0-9346-4db0-bdbc-1318669a6b1a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w\" (UID: \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:27:50.795603 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.795571 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtqr8\" (UniqueName: \"kubernetes.io/projected/9334b0a0-9346-4db0-bdbc-1318669a6b1a-kube-api-access-gtqr8\") pod \"payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w\" (UID: \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:27:50.812085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.812060 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx"] Sep 10 17:27:50.814824 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:27:50.814798 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaba8334_452f_4598_9110_764809181b0c.slice/crio-a1ce8fb979d453b851c57afdd085adc91e23b52e6761134f93eccf1201e6fd93 WatchSource:0}: Error finding container a1ce8fb979d453b851c57afdd085adc91e23b52e6761134f93eccf1201e6fd93: Status 404 returned error can't find the container with id a1ce8fb979d453b851c57afdd085adc91e23b52e6761134f93eccf1201e6fd93 Sep 10 17:27:50.989200 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:50.989167 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:27:51.088968 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:51.088934 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" event={"ID":"caba8334-452f-4598-9110-764809181b0c","Type":"ContainerStarted","Data":"a1ce8fb979d453b851c57afdd085adc91e23b52e6761134f93eccf1201e6fd93"} Sep 10 17:27:51.273949 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:51.273916 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w"] Sep 10 17:27:51.276406 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:27:51.276377 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9334b0a0_9346_4db0_bdbc_1318669a6b1a.slice/crio-b1f071003b8e10d24cce4580c45127fbe1886db54e7642652c69d7746b8e9fef WatchSource:0}: Error finding container b1f071003b8e10d24cce4580c45127fbe1886db54e7642652c69d7746b8e9fef: Status 404 returned error can't find the container with id b1f071003b8e10d24cce4580c45127fbe1886db54e7642652c69d7746b8e9fef Sep 10 17:27:52.096110 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:52.096075 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" event={"ID":"9334b0a0-9346-4db0-bdbc-1318669a6b1a","Type":"ContainerStarted","Data":"72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe"} Sep 10 17:27:52.096586 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:52.096228 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:27:52.096586 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:52.096279 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" event={"ID":"9334b0a0-9346-4db0-bdbc-1318669a6b1a","Type":"ContainerStarted","Data":"b1f071003b8e10d24cce4580c45127fbe1886db54e7642652c69d7746b8e9fef"} Sep 10 17:27:52.097628 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:52.097606 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" event={"ID":"caba8334-452f-4598-9110-764809181b0c","Type":"ContainerStarted","Data":"a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179"} Sep 10 17:27:52.097755 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:52.097653 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:52.117245 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:52.117198 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" podStartSLOduration=2.117183797 podStartE2EDuration="2.117183797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:52.115157377 +0000 UTC m=+871.213055738" watchObservedRunningTime="2026-09-10 17:27:52.117183797 +0000 UTC m=+871.215082158" Sep 10 17:27:52.134722 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:52.134663 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" podStartSLOduration=2.448060439 podStartE2EDuration="3.134644115s" totalImagesPullingTime="686.583676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:52.133869911 +0000 UTC m=+871.231768273" watchObservedRunningTime="2026-09-10 17:27:52.134644115 +0000 UTC m=+871.232542478" Sep 10 17:27:54.341393 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.341358 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf"] Sep 10 17:27:54.351660 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.351631 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.355895 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.355869 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e52f3775-openshift-default-dockercfg-cszv2\"" Sep 10 17:27:54.373394 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.373364 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf"] Sep 10 17:27:54.419185 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.419151 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-podinfo\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.419338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.419194 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-credential-socket\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.419338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.419212 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-envoy\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.419338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.419253 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-data\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.419338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.419309 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-socket\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.419338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.419327 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-certs\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.419526 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.419390 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-token\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.419526 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.419443 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istiod-ca-cert\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.419526 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.419457 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zn8\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-kube-api-access-j8zn8\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.520653 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.520616 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istiod-ca-cert\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.520653 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.520654 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zn8\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-kube-api-access-j8zn8\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.520909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.520676 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-podinfo\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.520909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.520701 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-credential-socket\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.520909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.520721 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-envoy\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.520909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.520748 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-data\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.520909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.520800 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-socket\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.520909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.520822 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-certs\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.520909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.520886 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-token\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.521462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.521213 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-credential-socket\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.521462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.521359 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-socket\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.521615 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.521588 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istiod-ca-cert\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.521805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.521783 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-certs\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.521880 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.521800 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-data\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.523383 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.523363 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-podinfo\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.523702 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.523686 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-envoy\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.539731 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.539697 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-token\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.542240 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.542201 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zn8\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-kube-api-access-j8zn8\") pod \"e2e-ait-e52f3775-openshift-default-85946d477b-swpgf\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.668027 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.667920 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:54.965005 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.964902 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf"] Sep 10 17:27:54.970764 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:27:54.970722 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4fc35b_b607_494e_8b68_53eeb7281a3d.slice/crio-14eeda9a54899e7699d8e2cb3c2ff523442e423fa096636bc3e5d5460998e750 WatchSource:0}: Error finding container 14eeda9a54899e7699d8e2cb3c2ff523442e423fa096636bc3e5d5460998e750: Status 404 returned error can't find the container with id 14eeda9a54899e7699d8e2cb3c2ff523442e423fa096636bc3e5d5460998e750 Sep 10 17:27:54.973971 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.973926 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:27:54.974110 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.974016 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:27:54.974110 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:54.974083 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:27:55.114596 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:55.114556 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" event={"ID":"cf4fc35b-b607-494e-8b68-53eeb7281a3d","Type":"ContainerStarted","Data":"60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed"} Sep 10 17:27:55.114735 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:55.114627 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" event={"ID":"cf4fc35b-b607-494e-8b68-53eeb7281a3d","Type":"ContainerStarted","Data":"14eeda9a54899e7699d8e2cb3c2ff523442e423fa096636bc3e5d5460998e750"} Sep 10 17:27:55.139671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:55.139616 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" podStartSLOduration=1.13959748 podStartE2EDuration="1.13959748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:55.138025962 +0000 UTC m=+874.235924323" watchObservedRunningTime="2026-09-10 17:27:55.13959748 +0000 UTC m=+874.237495842" Sep 10 17:27:55.669012 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:55.668970 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:56.674562 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:56.674524 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:57.125083 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:57.125049 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:57.125963 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:57.125945 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:27:58.108228 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.108198 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:27:58.137924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.137893 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp"] Sep 10 17:27:58.141575 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.141552 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:27:58.169056 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.168543 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-2da043-dockercfg-7l7fw\"" Sep 10 17:27:58.169056 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.168903 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-2da043\"" Sep 10 17:27:58.175642 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.175619 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp"] Sep 10 17:27:58.277056 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.276998 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw8kh\" (UniqueName: \"kubernetes.io/projected/5b92a5f0-6253-4825-aa14-70c1752c93d9-kube-api-access-gw8kh\") pod \"payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp\" (UID: \"5b92a5f0-6253-4825-aa14-70c1752c93d9\") " pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:27:58.277056 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.277062 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b92a5f0-6253-4825-aa14-70c1752c93d9-plugins-config-volume\") pod \"payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp\" (UID: \"5b92a5f0-6253-4825-aa14-70c1752c93d9\") " pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:27:58.378444 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.378352 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw8kh\" (UniqueName: \"kubernetes.io/projected/5b92a5f0-6253-4825-aa14-70c1752c93d9-kube-api-access-gw8kh\") pod \"payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp\" (UID: \"5b92a5f0-6253-4825-aa14-70c1752c93d9\") " pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:27:58.378444 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.378398 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b92a5f0-6253-4825-aa14-70c1752c93d9-plugins-config-volume\") pod \"payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp\" (UID: \"5b92a5f0-6253-4825-aa14-70c1752c93d9\") " pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:27:58.379231 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.379210 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b92a5f0-6253-4825-aa14-70c1752c93d9-plugins-config-volume\") pod \"payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp\" (UID: \"5b92a5f0-6253-4825-aa14-70c1752c93d9\") " pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:27:58.398717 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.398690 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw8kh\" (UniqueName: \"kubernetes.io/projected/5b92a5f0-6253-4825-aa14-70c1752c93d9-kube-api-access-gw8kh\") pod \"payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp\" (UID: \"5b92a5f0-6253-4825-aa14-70c1752c93d9\") " pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:27:58.476732 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.476691 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:27:58.769523 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:58.769485 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp"] Sep 10 17:27:58.776729 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:27:58.776636 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b92a5f0_6253_4825_aa14_70c1752c93d9.slice/crio-f7ee87fd8f1b7c187dc47562a4b0835ccafc2ff555b1724587153c93ecd41818 WatchSource:0}: Error finding container f7ee87fd8f1b7c187dc47562a4b0835ccafc2ff555b1724587153c93ecd41818: Status 404 returned error can't find the container with id f7ee87fd8f1b7c187dc47562a4b0835ccafc2ff555b1724587153c93ecd41818 Sep 10 17:27:59.136635 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.136604 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" event={"ID":"5b92a5f0-6253-4825-aa14-70c1752c93d9","Type":"ContainerStarted","Data":"4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20"} Sep 10 17:27:59.137080 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.136653 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" event={"ID":"5b92a5f0-6253-4825-aa14-70c1752c93d9","Type":"ContainerStarted","Data":"f7ee87fd8f1b7c187dc47562a4b0835ccafc2ff555b1724587153c93ecd41818"} Sep 10 17:27:59.137080 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.136847 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:27:59.162335 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.162270 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" podStartSLOduration=1.162246679 podStartE2EDuration="1.162246679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:59.161177615 +0000 UTC m=+878.259075976" watchObservedRunningTime="2026-09-10 17:27:59.162246679 +0000 UTC m=+878.260145040" Sep 10 17:27:59.623337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.623305 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw"] Sep 10 17:27:59.627558 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.627535 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:27:59.631229 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.631210 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e52f3775\"" Sep 10 17:27:59.631523 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.631292 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e52f3775-dockercfg-dnlpk\"" Sep 10 17:27:59.637678 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.637651 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw"] Sep 10 17:27:59.691974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.691812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxxj\" (UniqueName: \"kubernetes.io/projected/10373345-82eb-495f-aee6-cb6c849871ae-kube-api-access-8bxxj\") pod \"payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw\" (UID: \"10373345-82eb-495f-aee6-cb6c849871ae\") " pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:27:59.691974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.691880 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10373345-82eb-495f-aee6-cb6c849871ae-plugins-config-volume\") pod \"payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw\" (UID: \"10373345-82eb-495f-aee6-cb6c849871ae\") " pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:27:59.792788 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.792749 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxxj\" (UniqueName: \"kubernetes.io/projected/10373345-82eb-495f-aee6-cb6c849871ae-kube-api-access-8bxxj\") pod \"payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw\" (UID: \"10373345-82eb-495f-aee6-cb6c849871ae\") " pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:27:59.793000 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.792978 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10373345-82eb-495f-aee6-cb6c849871ae-plugins-config-volume\") pod \"payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw\" (UID: \"10373345-82eb-495f-aee6-cb6c849871ae\") " pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:27:59.793688 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.793660 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10373345-82eb-495f-aee6-cb6c849871ae-plugins-config-volume\") pod \"payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw\" (UID: \"10373345-82eb-495f-aee6-cb6c849871ae\") " pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:27:59.801666 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.801620 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxxj\" (UniqueName: \"kubernetes.io/projected/10373345-82eb-495f-aee6-cb6c849871ae-kube-api-access-8bxxj\") pod \"payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw\" (UID: \"10373345-82eb-495f-aee6-cb6c849871ae\") " pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:27:59.940761 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:27:59.940661 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:28:00.242114 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:00.242090 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw"] Sep 10 17:28:00.244074 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:28:00.244010 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10373345_82eb_495f_aee6_cb6c849871ae.slice/crio-491a03d7897520e577bb76c38b732cdbdf492e08c5c3aa80a02eec2473fa62a6 WatchSource:0}: Error finding container 491a03d7897520e577bb76c38b732cdbdf492e08c5c3aa80a02eec2473fa62a6: Status 404 returned error can't find the container with id 491a03d7897520e577bb76c38b732cdbdf492e08c5c3aa80a02eec2473fa62a6 Sep 10 17:28:01.149553 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:01.149513 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" event={"ID":"10373345-82eb-495f-aee6-cb6c849871ae","Type":"ContainerStarted","Data":"df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853"} Sep 10 17:28:01.149716 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:01.149577 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" event={"ID":"10373345-82eb-495f-aee6-cb6c849871ae","Type":"ContainerStarted","Data":"491a03d7897520e577bb76c38b732cdbdf492e08c5c3aa80a02eec2473fa62a6"} Sep 10 17:28:01.149906 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:01.149887 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:28:01.170949 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:01.170894 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" podStartSLOduration=2.170879234 podStartE2EDuration="2.170879234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:01.169073223 +0000 UTC m=+880.266971585" watchObservedRunningTime="2026-09-10 17:28:01.170879234 +0000 UTC m=+880.268777595" Sep 10 17:28:03.104697 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:03.104663 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:28:03.108189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:03.108154 2557 patch_prober.go:28] interesting pod/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 17:28:03.108300 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:03.108207 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 17:28:04.170233 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:04.170201 2557 generic.go:356] "Generic (PLEG): container finished" podID="a7360973-3aae-4c11-956b-348bd684e3c0" containerID="e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b" exitCode=0 Sep 10 17:28:04.170665 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:04.170258 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" event={"ID":"a7360973-3aae-4c11-956b-348bd684e3c0","Type":"ContainerDied","Data":"e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b"} Sep 10 17:28:04.170665 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:04.170657 2557 scope.go:117] "RemoveContainer" containerID="e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b" Sep 10 17:28:04.171894 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:04.171856 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:28:04.171998 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:04.171946 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:28:04.172085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:04.171994 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:28:05.186594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:05.186554 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" event={"ID":"a7360973-3aae-4c11-956b-348bd684e3c0","Type":"ContainerStarted","Data":"9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b"} Sep 10 17:28:05.811962 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:05.811922 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:28:05.812170 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:05.812147 2557 patch_prober.go:28] interesting pod/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 17:28:05.812222 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:05.812200 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 17:28:06.812903 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:06.812870 2557 patch_prober.go:28] interesting pod/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 17:28:06.813300 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:06.812928 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 17:28:07.812575 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:07.812543 2557 patch_prober.go:28] interesting pod/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 17:28:07.812770 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:07.812603 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 17:28:08.812791 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:08.812333 2557 patch_prober.go:28] interesting pod/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 17:28:08.812791 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:08.812402 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 17:28:09.812446 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:09.812408 2557 patch_prober.go:28] interesting pod/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 10 17:28:09.812615 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:09.812465 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 10 17:28:10.144312 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:10.144225 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:28:10.816280 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:10.816250 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:28:10.816577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:10.816556 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:28:10.817198 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:10.817178 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:28:12.161335 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:12.161275 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:28:13.626538 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.626498 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s"] Sep 10 17:28:13.629217 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.629199 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:13.632291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.632266 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-15ece2-dockercfg-qnl4b\"" Sep 10 17:28:13.632974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.632958 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-15ece2\"" Sep 10 17:28:13.652522 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.652488 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s"] Sep 10 17:28:13.696403 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.696375 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl"] Sep 10 17:28:13.700193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.700169 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:13.715326 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.715296 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl"] Sep 10 17:28:13.741336 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.741260 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s\" (UID: \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:13.741459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.741397 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmckx\" (UniqueName: \"kubernetes.io/projected/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-kube-api-access-qmckx\") pod \"payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s\" (UID: \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:13.842615 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.842573 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c3955d-45d3-4731-a9ed-77c2041c0380-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl\" (UID: \"20c3955d-45d3-4731-a9ed-77c2041c0380\") " pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:13.842794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.842628 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s\" (UID: \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:13.842794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.842677 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgsr4\" (UniqueName: \"kubernetes.io/projected/20c3955d-45d3-4731-a9ed-77c2041c0380-kube-api-access-fgsr4\") pod \"payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl\" (UID: \"20c3955d-45d3-4731-a9ed-77c2041c0380\") " pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:13.842794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.842704 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmckx\" (UniqueName: \"kubernetes.io/projected/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-kube-api-access-qmckx\") pod \"payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s\" (UID: \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:13.843510 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.843471 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s\" (UID: \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:13.862830 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.862789 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmckx\" (UniqueName: \"kubernetes.io/projected/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-kube-api-access-qmckx\") pod \"payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s\" (UID: \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:13.944372 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.944262 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c3955d-45d3-4731-a9ed-77c2041c0380-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl\" (UID: \"20c3955d-45d3-4731-a9ed-77c2041c0380\") " pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:13.944372 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.944340 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgsr4\" (UniqueName: \"kubernetes.io/projected/20c3955d-45d3-4731-a9ed-77c2041c0380-kube-api-access-fgsr4\") pod \"payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl\" (UID: \"20c3955d-45d3-4731-a9ed-77c2041c0380\") " pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:13.945340 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.945311 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c3955d-45d3-4731-a9ed-77c2041c0380-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl\" (UID: \"20c3955d-45d3-4731-a9ed-77c2041c0380\") " pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:13.953505 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.953467 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgsr4\" (UniqueName: \"kubernetes.io/projected/20c3955d-45d3-4731-a9ed-77c2041c0380-kube-api-access-fgsr4\") pod \"payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl\" (UID: \"20c3955d-45d3-4731-a9ed-77c2041c0380\") " pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:13.989275 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:13.989238 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:14.013386 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:14.013329 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:14.282146 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:14.282112 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s"] Sep 10 17:28:14.283093 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:28:14.283070 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1d8fa2_251d_45ed_bcbb_3a6ff24042dd.slice/crio-7b5f88ca38c172c7fada81ef7d9b8d4a42ee3a9f9edfaed923fa4aafae2a1250 WatchSource:0}: Error finding container 7b5f88ca38c172c7fada81ef7d9b8d4a42ee3a9f9edfaed923fa4aafae2a1250: Status 404 returned error can't find the container with id 7b5f88ca38c172c7fada81ef7d9b8d4a42ee3a9f9edfaed923fa4aafae2a1250 Sep 10 17:28:14.320472 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:14.320441 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl"] Sep 10 17:28:15.247927 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:15.247880 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" event={"ID":"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd","Type":"ContainerStarted","Data":"a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b"} Sep 10 17:28:15.248464 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:15.247954 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" event={"ID":"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd","Type":"ContainerStarted","Data":"7b5f88ca38c172c7fada81ef7d9b8d4a42ee3a9f9edfaed923fa4aafae2a1250"} Sep 10 17:28:15.248561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:15.248535 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:15.250167 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:15.250139 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" event={"ID":"20c3955d-45d3-4731-a9ed-77c2041c0380","Type":"ContainerStarted","Data":"34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96"} Sep 10 17:28:15.250304 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:15.250187 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" event={"ID":"20c3955d-45d3-4731-a9ed-77c2041c0380","Type":"ContainerStarted","Data":"8d41ab5059c46741d45ab75e05d2dc3eccda6ac15a9ce2684a55d203b5ae7545"} Sep 10 17:28:15.250304 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:15.250272 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:15.266825 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:15.266740 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" podStartSLOduration=2.26671827 podStartE2EDuration="2.26671827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:15.263397339 +0000 UTC m=+894.361295700" watchObservedRunningTime="2026-09-10 17:28:15.26671827 +0000 UTC m=+894.364616632" Sep 10 17:28:15.292968 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:15.292901 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" podStartSLOduration=2.292886707 podStartE2EDuration="2.292886707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:15.290253058 +0000 UTC m=+894.388151419" watchObservedRunningTime="2026-09-10 17:28:15.292886707 +0000 UTC m=+894.390785068" Sep 10 17:28:18.266434 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:18.266403 2557 generic.go:356] "Generic (PLEG): container finished" podID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerID="60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed" exitCode=0 Sep 10 17:28:18.267254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:18.266456 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" event={"ID":"cf4fc35b-b607-494e-8b68-53eeb7281a3d","Type":"ContainerDied","Data":"60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed"} Sep 10 17:28:18.267254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:18.266832 2557 scope.go:117] "RemoveContainer" containerID="60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed" Sep 10 17:28:18.267883 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:18.267852 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:28:18.267974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:18.267924 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:28:18.267974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:18.267958 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:28:19.276844 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:19.276796 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" event={"ID":"cf4fc35b-b607-494e-8b68-53eeb7281a3d","Type":"ContainerStarted","Data":"a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d"} Sep 10 17:28:19.668955 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:19.668852 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:28:19.669214 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:19.669188 2557 patch_prober.go:28] interesting pod/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 17:28:19.669293 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:19.669247 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 17:28:20.669601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:20.669559 2557 patch_prober.go:28] interesting pod/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 17:28:20.670229 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:20.670194 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 17:28:21.329725 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:21.329693 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:28:21.332672 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:21.332650 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:28:21.669116 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:21.668989 2557 patch_prober.go:28] interesting pod/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 17:28:21.669291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:21.669138 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 17:28:22.669309 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:22.669264 2557 patch_prober.go:28] interesting pod/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Sep 10 17:28:22.669811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:22.669349 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Sep 10 17:28:22.823092 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:22.823059 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw"] Sep 10 17:28:22.823375 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:22.823319 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" podUID="10373345-82eb-495f-aee6-cb6c849871ae" containerName="payload-processing" containerID="cri-o://df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853" gracePeriod=30 Sep 10 17:28:23.161379 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.161337 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:28:23.225446 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.209463 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf"] Sep 10 17:28:23.225952 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.225911 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" containerID="cri-o://a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d" gracePeriod=30 Sep 10 17:28:23.242945 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.242826 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bxxj\" (UniqueName: \"kubernetes.io/projected/10373345-82eb-495f-aee6-cb6c849871ae-kube-api-access-8bxxj\") pod \"10373345-82eb-495f-aee6-cb6c849871ae\" (UID: \"10373345-82eb-495f-aee6-cb6c849871ae\") " Sep 10 17:28:23.242945 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.242885 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10373345-82eb-495f-aee6-cb6c849871ae-plugins-config-volume\") pod \"10373345-82eb-495f-aee6-cb6c849871ae\" (UID: \"10373345-82eb-495f-aee6-cb6c849871ae\") " Sep 10 17:28:23.252326 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.247263 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10373345-82eb-495f-aee6-cb6c849871ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10373345-82eb-495f-aee6-cb6c849871ae" (UID: "10373345-82eb-495f-aee6-cb6c849871ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:28:23.256244 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.256203 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10373345-82eb-495f-aee6-cb6c849871ae-kube-api-access-8bxxj" (OuterVolumeSpecName: "kube-api-access-8bxxj") pod "10373345-82eb-495f-aee6-cb6c849871ae" (UID: "10373345-82eb-495f-aee6-cb6c849871ae"). InnerVolumeSpecName "kube-api-access-8bxxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:28:23.304018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.303983 2557 generic.go:356] "Generic (PLEG): container finished" podID="10373345-82eb-495f-aee6-cb6c849871ae" containerID="df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853" exitCode=0 Sep 10 17:28:23.304257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.304068 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" Sep 10 17:28:23.304257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.304112 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" event={"ID":"10373345-82eb-495f-aee6-cb6c849871ae","Type":"ContainerDied","Data":"df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853"} Sep 10 17:28:23.304257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.304176 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw" event={"ID":"10373345-82eb-495f-aee6-cb6c849871ae","Type":"ContainerDied","Data":"491a03d7897520e577bb76c38b732cdbdf492e08c5c3aa80a02eec2473fa62a6"} Sep 10 17:28:23.304257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.304198 2557 scope.go:117] "RemoveContainer" containerID="df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853" Sep 10 17:28:23.314370 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.314345 2557 scope.go:117] "RemoveContainer" containerID="df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853" Sep 10 17:28:23.314706 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:28:23.314680 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853\": container with ID starting with df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853 not found: ID does not exist" containerID="df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853" Sep 10 17:28:23.314788 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.314717 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853"} err="failed to get container status \"df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853\": rpc error: code = NotFound desc = could not find container \"df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853\": container with ID starting with df3654c3b4c997be4a79a33b7a7a55379d8e1b8ba10b52d68492b07790423853 not found: ID does not exist" Sep 10 17:28:23.330208 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.330186 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw"] Sep 10 17:28:23.339754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.339642 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e52f3775-6f8d9f8f87-mqlkw"] Sep 10 17:28:23.349756 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.349721 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bxxj\" (UniqueName: \"kubernetes.io/projected/10373345-82eb-495f-aee6-cb6c849871ae-kube-api-access-8bxxj\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:23.349756 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.349757 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10373345-82eb-495f-aee6-cb6c849871ae-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:23.393352 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:23.393313 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10373345-82eb-495f-aee6-cb6c849871ae" path="/var/lib/kubelet/pods/10373345-82eb-495f-aee6-cb6c849871ae/volumes" Sep 10 17:28:26.257884 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:26.257853 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:28:26.258367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:26.257931 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:28:27.125313 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:27.125274 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:28:30.210189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.210124 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238338 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-certs\") pod \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238382 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-credential-socket\") pod \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238439 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-data\") pod \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238477 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-podinfo\") pod \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238531 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-socket\") pod \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238615 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf4fc35b-b607-494e-8b68-53eeb7281a3d" (UID: "cf4fc35b-b607-494e-8b68-53eeb7281a3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238622 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-envoy\") pod \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238668 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8zn8\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-kube-api-access-j8zn8\") pod \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238694 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istiod-ca-cert\") pod \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " Sep 10 17:28:30.238711 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.238718 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-token\") pod \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\" (UID: \"cf4fc35b-b607-494e-8b68-53eeb7281a3d\") " Sep 10 17:28:30.239353 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.239018 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:30.239353 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.239322 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf4fc35b-b607-494e-8b68-53eeb7281a3d" (UID: "cf4fc35b-b607-494e-8b68-53eeb7281a3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:28:30.239598 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.239553 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf4fc35b-b607-494e-8b68-53eeb7281a3d" (UID: "cf4fc35b-b607-494e-8b68-53eeb7281a3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:28:30.240411 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.239866 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf4fc35b-b607-494e-8b68-53eeb7281a3d" (UID: "cf4fc35b-b607-494e-8b68-53eeb7281a3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:28:30.240411 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.240337 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf4fc35b-b607-494e-8b68-53eeb7281a3d" (UID: "cf4fc35b-b607-494e-8b68-53eeb7281a3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:28:30.242225 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.242187 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-kube-api-access-j8zn8" (OuterVolumeSpecName: "kube-api-access-j8zn8") pod "cf4fc35b-b607-494e-8b68-53eeb7281a3d" (UID: "cf4fc35b-b607-494e-8b68-53eeb7281a3d"). InnerVolumeSpecName "kube-api-access-j8zn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:28:30.242965 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.242942 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf4fc35b-b607-494e-8b68-53eeb7281a3d" (UID: "cf4fc35b-b607-494e-8b68-53eeb7281a3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:28:30.244682 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.244650 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf4fc35b-b607-494e-8b68-53eeb7281a3d" (UID: "cf4fc35b-b607-494e-8b68-53eeb7281a3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:28:30.251364 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.251338 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf4fc35b-b607-494e-8b68-53eeb7281a3d" (UID: "cf4fc35b-b607-494e-8b68-53eeb7281a3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:28:30.339459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.339428 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:30.339459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.339455 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:30.339459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.339464 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:30.339681 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.339473 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:30.339681 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.339482 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:30.339681 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.339491 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zn8\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-kube-api-access-j8zn8\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:30.339681 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.339499 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:30.339681 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.339508 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf4fc35b-b607-494e-8b68-53eeb7281a3d-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:28:30.345221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.345190 2557 generic.go:356] "Generic (PLEG): container finished" podID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerID="a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d" exitCode=0 Sep 10 17:28:30.345347 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.345246 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" Sep 10 17:28:30.345347 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.345270 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" event={"ID":"cf4fc35b-b607-494e-8b68-53eeb7281a3d","Type":"ContainerDied","Data":"a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d"} Sep 10 17:28:30.345347 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.345343 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf" event={"ID":"cf4fc35b-b607-494e-8b68-53eeb7281a3d","Type":"ContainerDied","Data":"14eeda9a54899e7699d8e2cb3c2ff523442e423fa096636bc3e5d5460998e750"} Sep 10 17:28:30.345463 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.345375 2557 scope.go:117] "RemoveContainer" containerID="a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d" Sep 10 17:28:30.355217 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.355192 2557 scope.go:117] "RemoveContainer" containerID="60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed" Sep 10 17:28:30.365283 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.365266 2557 scope.go:117] "RemoveContainer" containerID="a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d" Sep 10 17:28:30.365556 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:28:30.365538 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d\": container with ID starting with a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d not found: ID does not exist" containerID="a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d" Sep 10 17:28:30.365632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.365568 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d"} err="failed to get container status \"a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d\": rpc error: code = NotFound desc = could not find container \"a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d\": container with ID starting with a45a92bc0617a7229fe8cf310b83b91918bcb4bd9740f9e5ece1b1111ad4387d not found: ID does not exist" Sep 10 17:28:30.365632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.365592 2557 scope.go:117] "RemoveContainer" containerID="60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed" Sep 10 17:28:30.365896 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:28:30.365878 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed\": container with ID starting with 60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed not found: ID does not exist" containerID="60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed" Sep 10 17:28:30.365933 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.365903 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed"} err="failed to get container status \"60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed\": rpc error: code = NotFound desc = could not find container \"60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed\": container with ID starting with 60f08b3f2274a9f339c1f46e2adae5b4ccfa6c6a65aede5bff96fdb8ba8f0fed not found: ID does not exist" Sep 10 17:28:30.372464 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.372441 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf"] Sep 10 17:28:30.377285 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:30.377259 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e52f3775-openshift-default-85946d477b-swpgf"] Sep 10 17:28:31.096086 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096049 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt"] Sep 10 17:28:31.096528 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096509 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" Sep 10 17:28:31.096619 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096532 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" Sep 10 17:28:31.096619 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096558 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10373345-82eb-495f-aee6-cb6c849871ae" containerName="payload-processing" Sep 10 17:28:31.096619 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096568 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="10373345-82eb-495f-aee6-cb6c849871ae" containerName="payload-processing" Sep 10 17:28:31.096619 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096582 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" Sep 10 17:28:31.096619 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096591 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" Sep 10 17:28:31.096867 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096691 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="10373345-82eb-495f-aee6-cb6c849871ae" containerName="payload-processing" Sep 10 17:28:31.096867 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096704 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" Sep 10 17:28:31.096867 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.096713 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" containerName="istio-proxy" Sep 10 17:28:31.099555 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.099533 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:28:31.103567 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.103539 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2d96ccec\"" Sep 10 17:28:31.103749 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.103631 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2d96ccec-dockercfg-d4xcj\"" Sep 10 17:28:31.117725 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.117700 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt"] Sep 10 17:28:31.145919 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.145884 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4abe74d-d967-4ca2-b27d-aa061b478ed9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt\" (UID: \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:28:31.146302 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.146265 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzp6\" (UniqueName: \"kubernetes.io/projected/e4abe74d-d967-4ca2-b27d-aa061b478ed9-kube-api-access-lnzp6\") pod \"payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt\" (UID: \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:28:31.247352 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.247311 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4abe74d-d967-4ca2-b27d-aa061b478ed9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt\" (UID: \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:28:31.247713 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.247390 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzp6\" (UniqueName: \"kubernetes.io/projected/e4abe74d-d967-4ca2-b27d-aa061b478ed9-kube-api-access-lnzp6\") pod \"payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt\" (UID: \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:28:31.248137 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.248114 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4abe74d-d967-4ca2-b27d-aa061b478ed9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt\" (UID: \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:28:31.256606 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.256579 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzp6\" (UniqueName: \"kubernetes.io/projected/e4abe74d-d967-4ca2-b27d-aa061b478ed9-kube-api-access-lnzp6\") pod \"payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt\" (UID: \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:28:31.386754 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.386667 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf4fc35b-b607-494e-8b68-53eeb7281a3d" path="/var/lib/kubelet/pods/cf4fc35b-b607-494e-8b68-53eeb7281a3d/volumes" Sep 10 17:28:31.410563 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.410534 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:28:31.799594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:31.799537 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt"] Sep 10 17:28:31.815192 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:28:31.815156 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4abe74d_d967_4ca2_b27d_aa061b478ed9.slice/crio-994911da1aec15773bd98dfb166a11c485afe42fd63c0a3dbe0069044d3e5b8c WatchSource:0}: Error finding container 994911da1aec15773bd98dfb166a11c485afe42fd63c0a3dbe0069044d3e5b8c: Status 404 returned error can't find the container with id 994911da1aec15773bd98dfb166a11c485afe42fd63c0a3dbe0069044d3e5b8c Sep 10 17:28:32.358737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:32.358691 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" event={"ID":"e4abe74d-d967-4ca2-b27d-aa061b478ed9","Type":"ContainerStarted","Data":"e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe"} Sep 10 17:28:32.359198 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:32.358790 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" event={"ID":"e4abe74d-d967-4ca2-b27d-aa061b478ed9","Type":"ContainerStarted","Data":"994911da1aec15773bd98dfb166a11c485afe42fd63c0a3dbe0069044d3e5b8c"} Sep 10 17:28:32.359198 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:32.358836 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:28:32.376813 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:32.376765 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" podStartSLOduration=1.376749802 podStartE2EDuration="1.376749802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:28:32.374684561 +0000 UTC m=+911.472582922" watchObservedRunningTime="2026-09-10 17:28:32.376749802 +0000 UTC m=+911.474648164" Sep 10 17:28:43.366252 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:28:43.366216 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:29:09.711809 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:09.711774 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt"] Sep 10 17:29:09.712351 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:09.712000 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" podUID="e4abe74d-d967-4ca2-b27d-aa061b478ed9" containerName="payload-pre-processing" containerID="cri-o://e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe" gracePeriod=30 Sep 10 17:29:09.979841 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:09.979759 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:29:10.016451 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.016418 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4abe74d-d967-4ca2-b27d-aa061b478ed9-plugins-config-volume\") pod \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\" (UID: \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\") " Sep 10 17:29:10.016651 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.016476 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnzp6\" (UniqueName: \"kubernetes.io/projected/e4abe74d-d967-4ca2-b27d-aa061b478ed9-kube-api-access-lnzp6\") pod \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\" (UID: \"e4abe74d-d967-4ca2-b27d-aa061b478ed9\") " Sep 10 17:29:10.017176 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.017144 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4abe74d-d967-4ca2-b27d-aa061b478ed9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4abe74d-d967-4ca2-b27d-aa061b478ed9" (UID: "e4abe74d-d967-4ca2-b27d-aa061b478ed9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:29:10.019028 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.018999 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4abe74d-d967-4ca2-b27d-aa061b478ed9-kube-api-access-lnzp6" (OuterVolumeSpecName: "kube-api-access-lnzp6") pod "e4abe74d-d967-4ca2-b27d-aa061b478ed9" (UID: "e4abe74d-d967-4ca2-b27d-aa061b478ed9"). InnerVolumeSpecName "kube-api-access-lnzp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:29:10.117887 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.117847 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzp6\" (UniqueName: \"kubernetes.io/projected/e4abe74d-d967-4ca2-b27d-aa061b478ed9-kube-api-access-lnzp6\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:29:10.117887 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.117877 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4abe74d-d967-4ca2-b27d-aa061b478ed9-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:29:10.549495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.549450 2557 generic.go:356] "Generic (PLEG): container finished" podID="e4abe74d-d967-4ca2-b27d-aa061b478ed9" containerID="e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe" exitCode=0 Sep 10 17:29:10.549495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.549497 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" Sep 10 17:29:10.549719 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.549531 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" event={"ID":"e4abe74d-d967-4ca2-b27d-aa061b478ed9","Type":"ContainerDied","Data":"e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe"} Sep 10 17:29:10.549719 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.549581 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt" event={"ID":"e4abe74d-d967-4ca2-b27d-aa061b478ed9","Type":"ContainerDied","Data":"994911da1aec15773bd98dfb166a11c485afe42fd63c0a3dbe0069044d3e5b8c"} Sep 10 17:29:10.549719 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.549605 2557 scope.go:117] "RemoveContainer" containerID="e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe" Sep 10 17:29:10.563189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.563168 2557 scope.go:117] "RemoveContainer" containerID="e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe" Sep 10 17:29:10.563472 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:29:10.563453 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe\": container with ID starting with e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe not found: ID does not exist" containerID="e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe" Sep 10 17:29:10.563519 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.563490 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe"} err="failed to get container status \"e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe\": rpc error: code = NotFound desc = could not find container \"e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe\": container with ID starting with e4456e39570767ae2df0ac880563485fd5beb80fcc4ce9853ccbd1efdc7f6cbe not found: ID does not exist" Sep 10 17:29:10.576642 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.576618 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt"] Sep 10 17:29:10.585634 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:10.585607 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2d96ccec-54d84f5f4-6g4rt"] Sep 10 17:29:11.028459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.028427 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv"] Sep 10 17:29:11.029011 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.028990 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4abe74d-d967-4ca2-b27d-aa061b478ed9" containerName="payload-pre-processing" Sep 10 17:29:11.029134 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.029014 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4abe74d-d967-4ca2-b27d-aa061b478ed9" containerName="payload-pre-processing" Sep 10 17:29:11.029189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.029151 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4abe74d-d967-4ca2-b27d-aa061b478ed9" containerName="payload-pre-processing" Sep 10 17:29:11.031888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.031866 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.034394 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.034372 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e70241a1-openshift-default-dockercfg-nq955\"" Sep 10 17:29:11.056092 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.056064 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv"] Sep 10 17:29:11.127416 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.127375 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-credential-socket\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.127416 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.127425 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-token\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.127416 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.127447 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-envoy\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.127737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.127536 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-socket\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.127737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.127554 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-podinfo\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.127737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.127666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-certs\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.127737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.127710 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbxp\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-kube-api-access-cfbxp\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.127988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.127812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-data\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.127988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.127852 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istiod-ca-cert\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.228930 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.228871 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-data\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.228930 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.228923 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istiod-ca-cert\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.228957 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-credential-socket\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.228994 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-token\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.229022 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-envoy\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.229119 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-socket\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229472 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.229364 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-data\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229472 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.229142 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-podinfo\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229592 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.229479 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-certs\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229592 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.229511 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbxp\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-kube-api-access-cfbxp\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229851 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.229821 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istiod-ca-cert\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229954 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.229869 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-credential-socket\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.229954 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.229905 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-socket\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.230074 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.230053 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-certs\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.231701 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.231678 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-envoy\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.232273 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.232247 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-podinfo\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.237506 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.237475 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbxp\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-kube-api-access-cfbxp\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.240345 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.240326 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-token\") pod \"e2e-ait-e70241a1-openshift-default-7b57489694-27jvv\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.343143 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.343024 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:11.387783 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.387751 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4abe74d-d967-4ca2-b27d-aa061b478ed9" path="/var/lib/kubelet/pods/e4abe74d-d967-4ca2-b27d-aa061b478ed9/volumes" Sep 10 17:29:11.627919 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.627895 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv"] Sep 10 17:29:11.630287 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:29:11.630252 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bd0cf2_d14d_4fd6_9dda_3228c4866a3c.slice/crio-a35a694dea107518d98092d6bf033a4a10b05852cd5c5f96b12236bf517e6778 WatchSource:0}: Error finding container a35a694dea107518d98092d6bf033a4a10b05852cd5c5f96b12236bf517e6778: Status 404 returned error can't find the container with id a35a694dea107518d98092d6bf033a4a10b05852cd5c5f96b12236bf517e6778 Sep 10 17:29:11.632670 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.632624 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:11.632784 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.632717 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:11.632784 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:11.632762 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:12.562554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:12.562517 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" event={"ID":"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c","Type":"ContainerStarted","Data":"306ff1f5101f5070dbb733a855faac1bd390730bc8936fb5aa5b62f75a001d9c"} Sep 10 17:29:12.562936 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:12.562604 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" event={"ID":"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c","Type":"ContainerStarted","Data":"a35a694dea107518d98092d6bf033a4a10b05852cd5c5f96b12236bf517e6778"} Sep 10 17:29:12.600018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:12.599962 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" podStartSLOduration=1.5999456909999998 podStartE2EDuration="1.599945691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:12.598889785 +0000 UTC m=+951.696788146" watchObservedRunningTime="2026-09-10 17:29:12.599945691 +0000 UTC m=+951.697844051" Sep 10 17:29:13.343651 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:13.343610 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:13.350216 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:13.350188 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:13.568869 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:13.568832 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:13.570050 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:13.570011 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:33.675935 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:33.675903 2557 generic.go:356] "Generic (PLEG): container finished" podID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerID="306ff1f5101f5070dbb733a855faac1bd390730bc8936fb5aa5b62f75a001d9c" exitCode=0 Sep 10 17:29:33.676423 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:33.675985 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" event={"ID":"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c","Type":"ContainerDied","Data":"306ff1f5101f5070dbb733a855faac1bd390730bc8936fb5aa5b62f75a001d9c"} Sep 10 17:29:33.676467 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:33.676423 2557 scope.go:117] "RemoveContainer" containerID="306ff1f5101f5070dbb733a855faac1bd390730bc8936fb5aa5b62f75a001d9c" Sep 10 17:29:33.677527 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:33.677497 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:33.677617 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:33.677565 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:33.677617 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:33.677599 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:29:34.682920 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:34.682884 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" event={"ID":"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c","Type":"ContainerStarted","Data":"ac96044b484c417ce9d8860f0ae782ce6116800b044826344b68c1929c052c2e"} Sep 10 17:29:35.343875 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:35.343822 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:35.344126 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:35.343850 2557 patch_prober.go:28] interesting pod/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:29:35.344126 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:35.343997 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 17:29:36.343591 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:36.343551 2557 patch_prober.go:28] interesting pod/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:29:36.344968 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:36.344934 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 17:29:37.347069 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:37.345513 2557 patch_prober.go:28] interesting pod/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:29:37.347069 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:37.345594 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 17:29:38.343580 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:38.343540 2557 patch_prober.go:28] interesting pod/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:29:38.343885 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:38.343853 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 17:29:39.343405 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:39.343356 2557 patch_prober.go:28] interesting pod/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 10 17:29:39.343871 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:39.343428 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 10 17:29:40.347989 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:40.347953 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:40.348521 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:40.348496 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:40.349157 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:40.349136 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:29:57.705692 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:57.705607 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv"] Sep 10 17:29:57.706238 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:29:57.705846 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" containerID="cri-o://ac96044b484c417ce9d8860f0ae782ce6116800b044826344b68c1929c052c2e" gracePeriod=30 Sep 10 17:30:02.842110 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:02.842077 2557 generic.go:356] "Generic (PLEG): container finished" podID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerID="ac96044b484c417ce9d8860f0ae782ce6116800b044826344b68c1929c052c2e" exitCode=0 Sep 10 17:30:02.842495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:02.842157 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" event={"ID":"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c","Type":"ContainerDied","Data":"ac96044b484c417ce9d8860f0ae782ce6116800b044826344b68c1929c052c2e"} Sep 10 17:30:02.842495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:02.842274 2557 scope.go:117] "RemoveContainer" containerID="306ff1f5101f5070dbb733a855faac1bd390730bc8936fb5aa5b62f75a001d9c" Sep 10 17:30:02.975924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:02.975870 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:30:03.026718 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.026686 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-envoy\") pod \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " Sep 10 17:30:03.026879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.026747 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istiod-ca-cert\") pod \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " Sep 10 17:30:03.026879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.026767 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-data\") pod \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " Sep 10 17:30:03.026879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.026786 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-certs\") pod \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " Sep 10 17:30:03.026879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.026815 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-credential-socket\") pod \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " Sep 10 17:30:03.026879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.026869 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-socket\") pod \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " Sep 10 17:30:03.027170 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.026967 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-podinfo\") pod \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " Sep 10 17:30:03.027170 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.026994 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfbxp\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-kube-api-access-cfbxp\") pod \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " Sep 10 17:30:03.027170 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027027 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-token\") pod \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\" (UID: \"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c\") " Sep 10 17:30:03.027170 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027095 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" (UID: "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:30:03.027170 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027109 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-data" (OuterVolumeSpecName: "istio-data") pod "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" (UID: "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:30:03.027411 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027179 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" (UID: "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:30:03.027411 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027292 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" (UID: "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:30:03.027411 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027390 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" (UID: "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:30:03.027571 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027435 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:03.027571 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027457 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:03.027571 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027470 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:03.027571 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.027484 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:03.029425 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.029397 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" (UID: "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:30:03.029537 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.029431 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-token" (OuterVolumeSpecName: "istio-token") pod "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" (UID: "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:30:03.029537 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.029521 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" (UID: "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:30:03.029656 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.029637 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-kube-api-access-cfbxp" (OuterVolumeSpecName: "kube-api-access-cfbxp") pod "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" (UID: "48bd0cf2-d14d-4fd6-9dda-3228c4866a3c"). InnerVolumeSpecName "kube-api-access-cfbxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:30:03.127943 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.127909 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:03.127943 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.127934 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:03.127943 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.127944 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfbxp\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-kube-api-access-cfbxp\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:03.128185 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.127954 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:03.128185 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.127962 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:03.851284 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.851247 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" event={"ID":"48bd0cf2-d14d-4fd6-9dda-3228c4866a3c","Type":"ContainerDied","Data":"a35a694dea107518d98092d6bf033a4a10b05852cd5c5f96b12236bf517e6778"} Sep 10 17:30:03.851833 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.851320 2557 scope.go:117] "RemoveContainer" containerID="ac96044b484c417ce9d8860f0ae782ce6116800b044826344b68c1929c052c2e" Sep 10 17:30:03.851833 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.851263 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv" Sep 10 17:30:03.879798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.879742 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv"] Sep 10 17:30:03.882044 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:03.882009 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e70241a1-openshift-default-7b57489694-27jvv"] Sep 10 17:30:05.386713 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.386678 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" path="/var/lib/kubelet/pods/48bd0cf2-d14d-4fd6-9dda-3228c4866a3c/volumes" Sep 10 17:30:05.575798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.575759 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2"] Sep 10 17:30:05.576200 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.576173 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" Sep 10 17:30:05.576200 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.576202 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" Sep 10 17:30:05.576301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.576284 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" Sep 10 17:30:05.576301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.576291 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" Sep 10 17:30:05.578832 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.578815 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:05.581301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.581282 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-dcc1cd6c\"" Sep 10 17:30:05.608388 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.608356 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2"] Sep 10 17:30:05.653920 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.653846 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpnc\" (UniqueName: \"kubernetes.io/projected/654a6e34-2261-4aba-810b-472c31de4517-kube-api-access-xrpnc\") pod \"payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2\" (UID: \"654a6e34-2261-4aba-810b-472c31de4517\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:05.653920 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.653894 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a6e34-2261-4aba-810b-472c31de4517-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2\" (UID: \"654a6e34-2261-4aba-810b-472c31de4517\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:05.667729 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.667703 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4"] Sep 10 17:30:05.668111 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.668098 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" Sep 10 17:30:05.668163 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.668112 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="48bd0cf2-d14d-4fd6-9dda-3228c4866a3c" containerName="istio-proxy" Sep 10 17:30:05.670138 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.670121 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:05.674167 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.674148 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-dcc1cd6c-dockercfg-6rcbz\"" Sep 10 17:30:05.688410 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.688387 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4"] Sep 10 17:30:05.754784 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.754751 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpnc\" (UniqueName: \"kubernetes.io/projected/654a6e34-2261-4aba-810b-472c31de4517-kube-api-access-xrpnc\") pod \"payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2\" (UID: \"654a6e34-2261-4aba-810b-472c31de4517\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:05.754988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.754806 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25883f9-871b-4cd7-a6bc-a7e6529cba99-plugins-config-volume\") pod \"payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4\" (UID: \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\") " pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:05.754988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.754834 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2gdt\" (UniqueName: \"kubernetes.io/projected/b25883f9-871b-4cd7-a6bc-a7e6529cba99-kube-api-access-h2gdt\") pod \"payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4\" (UID: \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\") " pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:05.754988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.754925 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a6e34-2261-4aba-810b-472c31de4517-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2\" (UID: \"654a6e34-2261-4aba-810b-472c31de4517\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:05.755767 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.755743 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a6e34-2261-4aba-810b-472c31de4517-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2\" (UID: \"654a6e34-2261-4aba-810b-472c31de4517\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:05.764638 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.764614 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpnc\" (UniqueName: \"kubernetes.io/projected/654a6e34-2261-4aba-810b-472c31de4517-kube-api-access-xrpnc\") pod \"payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2\" (UID: \"654a6e34-2261-4aba-810b-472c31de4517\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:05.856308 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.856256 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25883f9-871b-4cd7-a6bc-a7e6529cba99-plugins-config-volume\") pod \"payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4\" (UID: \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\") " pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:05.856308 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.856309 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2gdt\" (UniqueName: \"kubernetes.io/projected/b25883f9-871b-4cd7-a6bc-a7e6529cba99-kube-api-access-h2gdt\") pod \"payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4\" (UID: \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\") " pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:05.857187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.857161 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25883f9-871b-4cd7-a6bc-a7e6529cba99-plugins-config-volume\") pod \"payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4\" (UID: \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\") " pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:05.867728 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.867695 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2gdt\" (UniqueName: \"kubernetes.io/projected/b25883f9-871b-4cd7-a6bc-a7e6529cba99-kube-api-access-h2gdt\") pod \"payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4\" (UID: \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\") " pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:05.888370 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.888341 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:05.979864 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:05.979835 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:06.186361 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.186336 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2"] Sep 10 17:30:06.188394 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:30:06.188365 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654a6e34_2261_4aba_810b_472c31de4517.slice/crio-80d1c80be7af79ba21dd170af9ae50b91dbb4c12aaf6ecfbd56abe16494666fb WatchSource:0}: Error finding container 80d1c80be7af79ba21dd170af9ae50b91dbb4c12aaf6ecfbd56abe16494666fb: Status 404 returned error can't find the container with id 80d1c80be7af79ba21dd170af9ae50b91dbb4c12aaf6ecfbd56abe16494666fb Sep 10 17:30:06.271433 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.271400 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4"] Sep 10 17:30:06.275575 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:30:06.275548 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25883f9_871b_4cd7_a6bc_a7e6529cba99.slice/crio-a48fbf1a1ba70bc87bac9efc5712658fb53b0e68d496ff7625160eae740c8e95 WatchSource:0}: Error finding container a48fbf1a1ba70bc87bac9efc5712658fb53b0e68d496ff7625160eae740c8e95: Status 404 returned error can't find the container with id a48fbf1a1ba70bc87bac9efc5712658fb53b0e68d496ff7625160eae740c8e95 Sep 10 17:30:06.872855 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.872816 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" event={"ID":"b25883f9-871b-4cd7-a6bc-a7e6529cba99","Type":"ContainerStarted","Data":"401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb"} Sep 10 17:30:06.873446 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.872885 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" event={"ID":"b25883f9-871b-4cd7-a6bc-a7e6529cba99","Type":"ContainerStarted","Data":"a48fbf1a1ba70bc87bac9efc5712658fb53b0e68d496ff7625160eae740c8e95"} Sep 10 17:30:06.873446 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.872988 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:06.874533 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.874505 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" event={"ID":"654a6e34-2261-4aba-810b-472c31de4517","Type":"ContainerStarted","Data":"b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682"} Sep 10 17:30:06.874645 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.874543 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" event={"ID":"654a6e34-2261-4aba-810b-472c31de4517","Type":"ContainerStarted","Data":"80d1c80be7af79ba21dd170af9ae50b91dbb4c12aaf6ecfbd56abe16494666fb"} Sep 10 17:30:06.874645 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.874586 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:06.903817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.903765 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" podStartSLOduration=1.903752074 podStartE2EDuration="1.903752074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:06.902752505 +0000 UTC m=+1006.000650866" watchObservedRunningTime="2026-09-10 17:30:06.903752074 +0000 UTC m=+1006.001650474" Sep 10 17:30:06.940346 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:06.940287 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" podStartSLOduration=1.940263745 podStartE2EDuration="1.940263745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:06.939298617 +0000 UTC m=+1006.037196978" watchObservedRunningTime="2026-09-10 17:30:06.940263745 +0000 UTC m=+1006.038162109" Sep 10 17:30:17.880776 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:17.880747 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:17.881268 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:17.881072 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:34.185837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:34.185796 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp"] Sep 10 17:30:34.186425 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:34.186082 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" podUID="5b92a5f0-6253-4825-aa14-70c1752c93d9" containerName="payload-processing" containerID="cri-o://4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20" gracePeriod=30 Sep 10 17:30:34.453360 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:34.453335 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:30:34.511495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:34.511463 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b92a5f0-6253-4825-aa14-70c1752c93d9-plugins-config-volume\") pod \"5b92a5f0-6253-4825-aa14-70c1752c93d9\" (UID: \"5b92a5f0-6253-4825-aa14-70c1752c93d9\") " Sep 10 17:30:34.511700 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:34.511527 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw8kh\" (UniqueName: \"kubernetes.io/projected/5b92a5f0-6253-4825-aa14-70c1752c93d9-kube-api-access-gw8kh\") pod \"5b92a5f0-6253-4825-aa14-70c1752c93d9\" (UID: \"5b92a5f0-6253-4825-aa14-70c1752c93d9\") " Sep 10 17:30:34.511889 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:34.511864 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b92a5f0-6253-4825-aa14-70c1752c93d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b92a5f0-6253-4825-aa14-70c1752c93d9" (UID: "5b92a5f0-6253-4825-aa14-70c1752c93d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:30:34.513549 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:34.513526 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b92a5f0-6253-4825-aa14-70c1752c93d9-kube-api-access-gw8kh" (OuterVolumeSpecName: "kube-api-access-gw8kh") pod "5b92a5f0-6253-4825-aa14-70c1752c93d9" (UID: "5b92a5f0-6253-4825-aa14-70c1752c93d9"). InnerVolumeSpecName "kube-api-access-gw8kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:30:34.612448 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:34.612415 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw8kh\" (UniqueName: \"kubernetes.io/projected/5b92a5f0-6253-4825-aa14-70c1752c93d9-kube-api-access-gw8kh\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:34.612448 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:34.612444 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b92a5f0-6253-4825-aa14-70c1752c93d9-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:35.013246 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.013213 2557 generic.go:356] "Generic (PLEG): container finished" podID="5b92a5f0-6253-4825-aa14-70c1752c93d9" containerID="4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20" exitCode=0 Sep 10 17:30:35.013418 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.013256 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" Sep 10 17:30:35.013418 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.013294 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" event={"ID":"5b92a5f0-6253-4825-aa14-70c1752c93d9","Type":"ContainerDied","Data":"4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20"} Sep 10 17:30:35.013418 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.013345 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp" event={"ID":"5b92a5f0-6253-4825-aa14-70c1752c93d9","Type":"ContainerDied","Data":"f7ee87fd8f1b7c187dc47562a4b0835ccafc2ff555b1724587153c93ecd41818"} Sep 10 17:30:35.013418 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.013367 2557 scope.go:117] "RemoveContainer" containerID="4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20" Sep 10 17:30:35.022424 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.022410 2557 scope.go:117] "RemoveContainer" containerID="4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20" Sep 10 17:30:35.022684 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:30:35.022666 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20\": container with ID starting with 4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20 not found: ID does not exist" containerID="4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20" Sep 10 17:30:35.022736 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.022695 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20"} err="failed to get container status \"4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20\": rpc error: code = NotFound desc = could not find container \"4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20\": container with ID starting with 4953ecd3679872934a5c1941676cbeedd58989338d6e22167aa255a5512c6e20 not found: ID does not exist" Sep 10 17:30:35.036798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.036771 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp"] Sep 10 17:30:35.041823 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.041789 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2da043-78db8c87bf-9wxvp"] Sep 10 17:30:35.388175 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:35.388086 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b92a5f0-6253-4825-aa14-70c1752c93d9" path="/var/lib/kubelet/pods/5b92a5f0-6253-4825-aa14-70c1752c93d9/volumes" Sep 10 17:30:43.169525 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.169482 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4"] Sep 10 17:30:43.169964 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.169771 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" podUID="b25883f9-871b-4cd7-a6bc-a7e6529cba99" containerName="payload-processing" containerID="cri-o://401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb" gracePeriod=30 Sep 10 17:30:43.192235 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.192207 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2"] Sep 10 17:30:43.193643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.192482 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" podUID="654a6e34-2261-4aba-810b-472c31de4517" containerName="payload-pre-processing" containerID="cri-o://b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682" gracePeriod=30 Sep 10 17:30:43.449954 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.449926 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:43.477585 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.477562 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:43.478869 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.478842 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25883f9-871b-4cd7-a6bc-a7e6529cba99-plugins-config-volume\") pod \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\" (UID: \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\") " Sep 10 17:30:43.478983 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.478916 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2gdt\" (UniqueName: \"kubernetes.io/projected/b25883f9-871b-4cd7-a6bc-a7e6529cba99-kube-api-access-h2gdt\") pod \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\" (UID: \"b25883f9-871b-4cd7-a6bc-a7e6529cba99\") " Sep 10 17:30:43.479201 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.479173 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25883f9-871b-4cd7-a6bc-a7e6529cba99-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b25883f9-871b-4cd7-a6bc-a7e6529cba99" (UID: "b25883f9-871b-4cd7-a6bc-a7e6529cba99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:30:43.479381 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.479354 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25883f9-871b-4cd7-a6bc-a7e6529cba99-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:43.481131 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.481110 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25883f9-871b-4cd7-a6bc-a7e6529cba99-kube-api-access-h2gdt" (OuterVolumeSpecName: "kube-api-access-h2gdt") pod "b25883f9-871b-4cd7-a6bc-a7e6529cba99" (UID: "b25883f9-871b-4cd7-a6bc-a7e6529cba99"). InnerVolumeSpecName "kube-api-access-h2gdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:30:43.580675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.580644 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrpnc\" (UniqueName: \"kubernetes.io/projected/654a6e34-2261-4aba-810b-472c31de4517-kube-api-access-xrpnc\") pod \"654a6e34-2261-4aba-810b-472c31de4517\" (UID: \"654a6e34-2261-4aba-810b-472c31de4517\") " Sep 10 17:30:43.580855 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.580744 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a6e34-2261-4aba-810b-472c31de4517-plugins-config-volume\") pod \"654a6e34-2261-4aba-810b-472c31de4517\" (UID: \"654a6e34-2261-4aba-810b-472c31de4517\") " Sep 10 17:30:43.581010 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.580995 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2gdt\" (UniqueName: \"kubernetes.io/projected/b25883f9-871b-4cd7-a6bc-a7e6529cba99-kube-api-access-h2gdt\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:43.581138 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.581114 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654a6e34-2261-4aba-810b-472c31de4517-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "654a6e34-2261-4aba-810b-472c31de4517" (UID: "654a6e34-2261-4aba-810b-472c31de4517"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:30:43.582767 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.582742 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654a6e34-2261-4aba-810b-472c31de4517-kube-api-access-xrpnc" (OuterVolumeSpecName: "kube-api-access-xrpnc") pod "654a6e34-2261-4aba-810b-472c31de4517" (UID: "654a6e34-2261-4aba-810b-472c31de4517"). InnerVolumeSpecName "kube-api-access-xrpnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:30:43.681908 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.681793 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrpnc\" (UniqueName: \"kubernetes.io/projected/654a6e34-2261-4aba-810b-472c31de4517-kube-api-access-xrpnc\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:43.681908 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:43.681842 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a6e34-2261-4aba-810b-472c31de4517-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:30:44.060247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.060211 2557 generic.go:356] "Generic (PLEG): container finished" podID="b25883f9-871b-4cd7-a6bc-a7e6529cba99" containerID="401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb" exitCode=0 Sep 10 17:30:44.060443 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.060266 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" Sep 10 17:30:44.060443 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.060292 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" event={"ID":"b25883f9-871b-4cd7-a6bc-a7e6529cba99","Type":"ContainerDied","Data":"401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb"} Sep 10 17:30:44.060443 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.060344 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4" event={"ID":"b25883f9-871b-4cd7-a6bc-a7e6529cba99","Type":"ContainerDied","Data":"a48fbf1a1ba70bc87bac9efc5712658fb53b0e68d496ff7625160eae740c8e95"} Sep 10 17:30:44.060443 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.060366 2557 scope.go:117] "RemoveContainer" containerID="401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb" Sep 10 17:30:44.061574 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.061555 2557 generic.go:356] "Generic (PLEG): container finished" podID="654a6e34-2261-4aba-810b-472c31de4517" containerID="b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682" exitCode=0 Sep 10 17:30:44.061649 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.061592 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" Sep 10 17:30:44.061710 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.061645 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" event={"ID":"654a6e34-2261-4aba-810b-472c31de4517","Type":"ContainerDied","Data":"b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682"} Sep 10 17:30:44.061710 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.061695 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2" event={"ID":"654a6e34-2261-4aba-810b-472c31de4517","Type":"ContainerDied","Data":"80d1c80be7af79ba21dd170af9ae50b91dbb4c12aaf6ecfbd56abe16494666fb"} Sep 10 17:30:44.070096 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.069969 2557 scope.go:117] "RemoveContainer" containerID="401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb" Sep 10 17:30:44.070335 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:30:44.070312 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb\": container with ID starting with 401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb not found: ID does not exist" containerID="401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb" Sep 10 17:30:44.070449 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.070344 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb"} err="failed to get container status \"401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb\": rpc error: code = NotFound desc = could not find container \"401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb\": container with ID starting with 401099c42d3f90fa7217bff9338806ec734dfc6b229dae5c7989f91be7947feb not found: ID does not exist" Sep 10 17:30:44.070449 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.070368 2557 scope.go:117] "RemoveContainer" containerID="b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682" Sep 10 17:30:44.078199 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.078184 2557 scope.go:117] "RemoveContainer" containerID="b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682" Sep 10 17:30:44.078473 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:30:44.078455 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682\": container with ID starting with b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682 not found: ID does not exist" containerID="b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682" Sep 10 17:30:44.078527 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.078480 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682"} err="failed to get container status \"b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682\": rpc error: code = NotFound desc = could not find container \"b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682\": container with ID starting with b414e4c37a2b30f93708a96c93ffc2dd504e4ffc1566269dee0a12ceb3100682 not found: ID does not exist" Sep 10 17:30:44.089414 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.089392 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2"] Sep 10 17:30:44.094747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.094724 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-dcc1cd6c-695dd984f4-ccxq2"] Sep 10 17:30:44.108643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.108622 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4"] Sep 10 17:30:44.115489 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:44.115469 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-dcc1cd6c-d74cd6485-tb6w4"] Sep 10 17:30:45.386684 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:45.386650 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="654a6e34-2261-4aba-810b-472c31de4517" path="/var/lib/kubelet/pods/654a6e34-2261-4aba-810b-472c31de4517/volumes" Sep 10 17:30:45.387059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:45.387017 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25883f9-871b-4cd7-a6bc-a7e6529cba99" path="/var/lib/kubelet/pods/b25883f9-871b-4cd7-a6bc-a7e6529cba99/volumes" Sep 10 17:30:51.957451 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957415 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6"] Sep 10 17:30:51.957834 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957780 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="654a6e34-2261-4aba-810b-472c31de4517" containerName="payload-pre-processing" Sep 10 17:30:51.957834 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957792 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="654a6e34-2261-4aba-810b-472c31de4517" containerName="payload-pre-processing" Sep 10 17:30:51.957834 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957816 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25883f9-871b-4cd7-a6bc-a7e6529cba99" containerName="payload-processing" Sep 10 17:30:51.957834 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957822 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25883f9-871b-4cd7-a6bc-a7e6529cba99" containerName="payload-processing" Sep 10 17:30:51.957834 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957836 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b92a5f0-6253-4825-aa14-70c1752c93d9" containerName="payload-processing" Sep 10 17:30:51.958003 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957842 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b92a5f0-6253-4825-aa14-70c1752c93d9" containerName="payload-processing" Sep 10 17:30:51.958003 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957905 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b92a5f0-6253-4825-aa14-70c1752c93d9" containerName="payload-processing" Sep 10 17:30:51.958003 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957913 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25883f9-871b-4cd7-a6bc-a7e6529cba99" containerName="payload-processing" Sep 10 17:30:51.958003 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.957922 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="654a6e34-2261-4aba-810b-472c31de4517" containerName="payload-pre-processing" Sep 10 17:30:51.959818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.959802 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:30:51.964017 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.963990 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-9e9c14-dockercfg-h7kgb\"" Sep 10 17:30:51.964191 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.964112 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9e9c14\"" Sep 10 17:30:51.977701 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:51.977670 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6"] Sep 10 17:30:52.054901 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:52.054865 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c3288-f6a5-4a31-8f14-45f90cb15abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6\" (UID: \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:30:52.055097 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:52.054931 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v49g\" (UniqueName: \"kubernetes.io/projected/da6c3288-f6a5-4a31-8f14-45f90cb15abe-kube-api-access-9v49g\") pod \"payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6\" (UID: \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:30:52.156186 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:52.156151 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c3288-f6a5-4a31-8f14-45f90cb15abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6\" (UID: \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:30:52.156352 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:52.156209 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v49g\" (UniqueName: \"kubernetes.io/projected/da6c3288-f6a5-4a31-8f14-45f90cb15abe-kube-api-access-9v49g\") pod \"payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6\" (UID: \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:30:52.156860 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:52.156838 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c3288-f6a5-4a31-8f14-45f90cb15abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6\" (UID: \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:30:52.166497 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:52.166468 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v49g\" (UniqueName: \"kubernetes.io/projected/da6c3288-f6a5-4a31-8f14-45f90cb15abe-kube-api-access-9v49g\") pod \"payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6\" (UID: \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:30:52.270411 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:52.270368 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:30:52.549713 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:52.549680 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6"] Sep 10 17:30:52.551373 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:30:52.551349 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6c3288_f6a5_4a31_8f14_45f90cb15abe.slice/crio-42d60558c94e68020573c999b10ba938ddbb86a32806d9b6bbcc9d647e534853 WatchSource:0}: Error finding container 42d60558c94e68020573c999b10ba938ddbb86a32806d9b6bbcc9d647e534853: Status 404 returned error can't find the container with id 42d60558c94e68020573c999b10ba938ddbb86a32806d9b6bbcc9d647e534853 Sep 10 17:30:53.114557 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:53.114525 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" event={"ID":"da6c3288-f6a5-4a31-8f14-45f90cb15abe","Type":"ContainerStarted","Data":"77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967"} Sep 10 17:30:53.114959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:53.114574 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" event={"ID":"da6c3288-f6a5-4a31-8f14-45f90cb15abe","Type":"ContainerStarted","Data":"42d60558c94e68020573c999b10ba938ddbb86a32806d9b6bbcc9d647e534853"} Sep 10 17:30:53.114959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:53.114638 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:30:53.142546 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:30:53.142495 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" podStartSLOduration=2.14247818 podStartE2EDuration="2.14247818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:53.141451238 +0000 UTC m=+1052.239349599" watchObservedRunningTime="2026-09-10 17:30:53.14247818 +0000 UTC m=+1052.240376540" Sep 10 17:31:04.122747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:31:04.122714 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:32:22.694518 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:22.694472 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6"] Sep 10 17:32:22.695121 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:22.694710 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" podUID="da6c3288-f6a5-4a31-8f14-45f90cb15abe" containerName="payload-pre-processing" containerID="cri-o://77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967" gracePeriod=30 Sep 10 17:32:22.953731 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:22.953668 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:32:23.070568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.070533 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c3288-f6a5-4a31-8f14-45f90cb15abe-plugins-config-volume\") pod \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\" (UID: \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\") " Sep 10 17:32:23.070763 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.070614 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v49g\" (UniqueName: \"kubernetes.io/projected/da6c3288-f6a5-4a31-8f14-45f90cb15abe-kube-api-access-9v49g\") pod \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\" (UID: \"da6c3288-f6a5-4a31-8f14-45f90cb15abe\") " Sep 10 17:32:23.070902 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.070879 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6c3288-f6a5-4a31-8f14-45f90cb15abe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da6c3288-f6a5-4a31-8f14-45f90cb15abe" (UID: "da6c3288-f6a5-4a31-8f14-45f90cb15abe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:32:23.072823 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.072786 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6c3288-f6a5-4a31-8f14-45f90cb15abe-kube-api-access-9v49g" (OuterVolumeSpecName: "kube-api-access-9v49g") pod "da6c3288-f6a5-4a31-8f14-45f90cb15abe" (UID: "da6c3288-f6a5-4a31-8f14-45f90cb15abe"). InnerVolumeSpecName "kube-api-access-9v49g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:32:23.172103 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.172071 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v49g\" (UniqueName: \"kubernetes.io/projected/da6c3288-f6a5-4a31-8f14-45f90cb15abe-kube-api-access-9v49g\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:32:23.172103 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.172097 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6c3288-f6a5-4a31-8f14-45f90cb15abe-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:32:23.554380 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.554336 2557 generic.go:356] "Generic (PLEG): container finished" podID="da6c3288-f6a5-4a31-8f14-45f90cb15abe" containerID="77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967" exitCode=0 Sep 10 17:32:23.554380 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.554381 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" Sep 10 17:32:23.554601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.554425 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" event={"ID":"da6c3288-f6a5-4a31-8f14-45f90cb15abe","Type":"ContainerDied","Data":"77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967"} Sep 10 17:32:23.554601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.554478 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6" event={"ID":"da6c3288-f6a5-4a31-8f14-45f90cb15abe","Type":"ContainerDied","Data":"42d60558c94e68020573c999b10ba938ddbb86a32806d9b6bbcc9d647e534853"} Sep 10 17:32:23.554601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.554504 2557 scope.go:117] "RemoveContainer" containerID="77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967" Sep 10 17:32:23.563025 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.563008 2557 scope.go:117] "RemoveContainer" containerID="77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967" Sep 10 17:32:23.563398 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:32:23.563378 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967\": container with ID starting with 77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967 not found: ID does not exist" containerID="77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967" Sep 10 17:32:23.563463 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.563406 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967"} err="failed to get container status \"77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967\": rpc error: code = NotFound desc = could not find container \"77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967\": container with ID starting with 77cf77d28efe2285ae5d636909bef5eeb813b700b5bebab4d3fdc695ef47d967 not found: ID does not exist" Sep 10 17:32:23.571410 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.571389 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6"] Sep 10 17:32:23.577241 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:23.577219 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-9e9c14-66f6f7dd46-frkv6"] Sep 10 17:32:25.251873 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.251843 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br"] Sep 10 17:32:25.252347 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.252330 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6c3288-f6a5-4a31-8f14-45f90cb15abe" containerName="payload-pre-processing" Sep 10 17:32:25.252391 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.252350 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6c3288-f6a5-4a31-8f14-45f90cb15abe" containerName="payload-pre-processing" Sep 10 17:32:25.252426 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.252421 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6c3288-f6a5-4a31-8f14-45f90cb15abe" containerName="payload-pre-processing" Sep 10 17:32:25.256779 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.256755 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.262000 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.261970 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-851d1193-openshift-default-dockercfg-l6q2k\"" Sep 10 17:32:25.275928 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.275903 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br"] Sep 10 17:32:25.290139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.290112 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-credential-socket\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.290291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.290152 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-token\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.290291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.290174 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hct9\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-kube-api-access-7hct9\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.290291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.290275 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-certs\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.290466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.290343 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-socket\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.290466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.290378 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-data\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.290466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.290453 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-envoy\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.290663 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.290483 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-podinfo\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.290663 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.290517 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132308f6-43b1-48ea-8de1-effceb49f3f0-istiod-ca-cert\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.387714 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.387680 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6c3288-f6a5-4a31-8f14-45f90cb15abe" path="/var/lib/kubelet/pods/da6c3288-f6a5-4a31-8f14-45f90cb15abe/volumes" Sep 10 17:32:25.390980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.390956 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-envoy\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391148 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.390984 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-podinfo\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391148 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132308f6-43b1-48ea-8de1-effceb49f3f0-istiod-ca-cert\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391148 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391091 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-credential-socket\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391148 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391124 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-token\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391374 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391282 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hct9\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-kube-api-access-7hct9\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391374 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391329 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-certs\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391481 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391397 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-socket\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391481 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391429 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-data\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391587 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391510 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-credential-socket\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391829 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391787 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-data\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391829 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391818 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-certs\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391992 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391907 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-socket\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.391992 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.391926 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132308f6-43b1-48ea-8de1-effceb49f3f0-istiod-ca-cert\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.394077 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.394028 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-envoy\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.394182 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.394164 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-podinfo\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.400496 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.400470 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-token\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.400590 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.400565 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hct9\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-kube-api-access-7hct9\") pod \"e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.570394 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.570115 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:25.851143 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.851066 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br"] Sep 10 17:32:25.854855 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:32:25.854820 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132308f6_43b1_48ea_8de1_effceb49f3f0.slice/crio-94559c0708928f0d66d15e75f2721fa39dde87e058f831f478d98a0baa50f6d9 WatchSource:0}: Error finding container 94559c0708928f0d66d15e75f2721fa39dde87e058f831f478d98a0baa50f6d9: Status 404 returned error can't find the container with id 94559c0708928f0d66d15e75f2721fa39dde87e058f831f478d98a0baa50f6d9 Sep 10 17:32:25.857432 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.857395 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:32:25.857532 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.857480 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:32:25.857532 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:25.857526 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:32:26.578211 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:26.578177 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" event={"ID":"132308f6-43b1-48ea-8de1-effceb49f3f0","Type":"ContainerStarted","Data":"1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310"} Sep 10 17:32:26.578621 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:26.578228 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" event={"ID":"132308f6-43b1-48ea-8de1-effceb49f3f0","Type":"ContainerStarted","Data":"94559c0708928f0d66d15e75f2721fa39dde87e058f831f478d98a0baa50f6d9"} Sep 10 17:32:26.599562 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:26.599509 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" podStartSLOduration=1.599491832 podStartE2EDuration="1.599491832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:26.598207866 +0000 UTC m=+1145.696106228" watchObservedRunningTime="2026-09-10 17:32:26.599491832 +0000 UTC m=+1145.697390192" Sep 10 17:32:27.570676 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:27.570638 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:27.576254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:27.576229 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:27.582611 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:27.582585 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:27.583276 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:27.583255 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:31.568064 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.568008 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl"] Sep 10 17:32:31.572145 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.572118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.576097 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.576076 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-8bd101-openshift-default-dockercfg-ccv8z\"" Sep 10 17:32:31.592735 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.592705 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl"] Sep 10 17:32:31.652416 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.652374 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-socket\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.652602 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.652471 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-credential-socket\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.652602 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.652508 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64635bf2-6766-4267-9851-f4a4f53a8bfb-istiod-ca-cert\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.652602 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.652530 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-certs\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.652753 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.652625 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-data\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.652753 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.652663 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvg99\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-kube-api-access-jvg99\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.652753 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.652723 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-podinfo\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.652753 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.652760 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-token\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.652970 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.652810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-envoy\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.754373 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.754337 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-envoy\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.754567 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.754394 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-socket\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.754567 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.754420 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-credential-socket\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.754567 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.754437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64635bf2-6766-4267-9851-f4a4f53a8bfb-istiod-ca-cert\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.754839 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.754808 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-socket\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.755018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.754826 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-certs\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.755152 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.755125 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-certs\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.755246 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.754919 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-credential-socket\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.755313 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.755252 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64635bf2-6766-4267-9851-f4a4f53a8bfb-istiod-ca-cert\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.755373 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.755360 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-data\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.755428 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.755393 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvg99\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-kube-api-access-jvg99\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.755497 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.755480 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-podinfo\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.755559 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.755517 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-token\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.755623 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.755604 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-data\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.756884 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.756856 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-envoy\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.758133 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.758106 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-podinfo\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.764658 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.764600 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-token\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.765585 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.765554 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvg99\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-kube-api-access-jvg99\") pod \"e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.815215 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.815100 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767"] Sep 10 17:32:31.820719 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.820633 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:31.824238 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.824207 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-851d1193\"" Sep 10 17:32:31.824923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.824880 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-851d1193-dockercfg-vl5nq\"" Sep 10 17:32:31.838856 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.838825 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767"] Sep 10 17:32:31.886705 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.886672 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:31.957758 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.957709 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mptl\" (UniqueName: \"kubernetes.io/projected/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-kube-api-access-2mptl\") pod \"payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767\" (UID: \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\") " pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:31.957945 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:31.957835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-plugins-config-volume\") pod \"payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767\" (UID: \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\") " pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:32.058923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.058885 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mptl\" (UniqueName: \"kubernetes.io/projected/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-kube-api-access-2mptl\") pod \"payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767\" (UID: \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\") " pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:32.059117 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.058961 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-plugins-config-volume\") pod \"payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767\" (UID: \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\") " pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:32.059785 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.059761 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-plugins-config-volume\") pod \"payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767\" (UID: \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\") " pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:32.066691 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.066667 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mptl\" (UniqueName: \"kubernetes.io/projected/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-kube-api-access-2mptl\") pod \"payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767\" (UID: \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\") " pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:32.145873 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.145779 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:32.183738 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.183710 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl"] Sep 10 17:32:32.186147 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:32:32.186112 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64635bf2_6766_4267_9851_f4a4f53a8bfb.slice/crio-52b79f76b32d5b00abc5efcac2456e8da34e8d45a039e1015b3abfb03ddf6c0a WatchSource:0}: Error finding container 52b79f76b32d5b00abc5efcac2456e8da34e8d45a039e1015b3abfb03ddf6c0a: Status 404 returned error can't find the container with id 52b79f76b32d5b00abc5efcac2456e8da34e8d45a039e1015b3abfb03ddf6c0a Sep 10 17:32:32.189304 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.189269 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:32:32.189413 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.189339 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:32:32.189413 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.189370 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:32:32.425287 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.425260 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767"] Sep 10 17:32:32.427866 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:32:32.427843 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d7662a_ed5f_4ee6_ae71_2e448222a4bf.slice/crio-19a8aae84c1b5e4fecf0da9d931af96b5cd034a65c254a342376e7d458d4b4c6 WatchSource:0}: Error finding container 19a8aae84c1b5e4fecf0da9d931af96b5cd034a65c254a342376e7d458d4b4c6: Status 404 returned error can't find the container with id 19a8aae84c1b5e4fecf0da9d931af96b5cd034a65c254a342376e7d458d4b4c6 Sep 10 17:32:32.608271 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.608228 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" event={"ID":"64635bf2-6766-4267-9851-f4a4f53a8bfb","Type":"ContainerStarted","Data":"5dc6f7a4f4fe184b4aad7e809bbe10477aec01b2ccf9b6b467c721d4abcb7a63"} Sep 10 17:32:32.608739 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.608292 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" event={"ID":"64635bf2-6766-4267-9851-f4a4f53a8bfb","Type":"ContainerStarted","Data":"52b79f76b32d5b00abc5efcac2456e8da34e8d45a039e1015b3abfb03ddf6c0a"} Sep 10 17:32:32.609844 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.609822 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" event={"ID":"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf","Type":"ContainerStarted","Data":"50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132"} Sep 10 17:32:32.609947 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.609858 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" event={"ID":"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf","Type":"ContainerStarted","Data":"19a8aae84c1b5e4fecf0da9d931af96b5cd034a65c254a342376e7d458d4b4c6"} Sep 10 17:32:32.609994 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.609949 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:32.631302 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.631237 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" podStartSLOduration=1.6312165699999999 podStartE2EDuration="1.63121657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:32.628008421 +0000 UTC m=+1151.725906782" watchObservedRunningTime="2026-09-10 17:32:32.63121657 +0000 UTC m=+1151.729114932" Sep 10 17:32:32.648573 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.648507 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" podStartSLOduration=1.648482599 podStartE2EDuration="1.648482599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:32.646385606 +0000 UTC m=+1151.744283969" watchObservedRunningTime="2026-09-10 17:32:32.648482599 +0000 UTC m=+1151.746380962" Sep 10 17:32:32.886792 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:32.886758 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:33.893152 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:33.893116 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:34.620267 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:34.620233 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:34.621221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:34.621202 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:32:43.617496 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:43.617464 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:32:48.718847 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:48.718810 2557 generic.go:356] "Generic (PLEG): container finished" podID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerID="1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310" exitCode=0 Sep 10 17:32:48.719271 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:48.718883 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" event={"ID":"132308f6-43b1-48ea-8de1-effceb49f3f0","Type":"ContainerDied","Data":"1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310"} Sep 10 17:32:48.719338 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:48.719324 2557 scope.go:117] "RemoveContainer" containerID="1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310" Sep 10 17:32:48.720432 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:48.720402 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:32:48.720506 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:48.720479 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:32:48.720549 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:48.720512 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:32:49.726122 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:49.726088 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" event={"ID":"132308f6-43b1-48ea-8de1-effceb49f3f0","Type":"ContainerStarted","Data":"37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69"} Sep 10 17:32:50.570833 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:50.570798 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:50.571095 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:50.571071 2557 patch_prober.go:28] interesting pod/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 17:32:50.571167 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:50.571124 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 17:32:51.571587 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:51.571544 2557 patch_prober.go:28] interesting pod/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 17:32:51.571983 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:51.571614 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 17:32:52.571350 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:52.571303 2557 patch_prober.go:28] interesting pod/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 17:32:52.571561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:52.571376 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 17:32:53.570671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:53.570633 2557 patch_prober.go:28] interesting pod/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 17:32:53.571097 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:53.570742 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 17:32:54.571216 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:54.571173 2557 patch_prober.go:28] interesting pod/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 10 17:32:54.571612 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:54.571234 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 10 17:32:55.575808 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:55.575779 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:55.576320 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:55.576226 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:32:55.576812 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:32:55.576787 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:33:11.485049 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.484993 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767"] Sep 10 17:33:11.485666 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.485298 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" podUID="e0d7662a-ed5f-4ee6-ae71-2e448222a4bf" containerName="payload-processing" containerID="cri-o://50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132" gracePeriod=30 Sep 10 17:33:11.758282 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.758255 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:33:11.848436 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.848378 2557 generic.go:356] "Generic (PLEG): container finished" podID="e0d7662a-ed5f-4ee6-ae71-2e448222a4bf" containerID="50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132" exitCode=0 Sep 10 17:33:11.848638 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.848455 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" event={"ID":"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf","Type":"ContainerDied","Data":"50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132"} Sep 10 17:33:11.848638 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.848505 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" event={"ID":"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf","Type":"ContainerDied","Data":"19a8aae84c1b5e4fecf0da9d931af96b5cd034a65c254a342376e7d458d4b4c6"} Sep 10 17:33:11.848638 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.848567 2557 scope.go:117] "RemoveContainer" containerID="50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132" Sep 10 17:33:11.848816 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.848697 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767" Sep 10 17:33:11.852237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.852209 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mptl\" (UniqueName: \"kubernetes.io/projected/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-kube-api-access-2mptl\") pod \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\" (UID: \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\") " Sep 10 17:33:11.852474 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.852438 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-plugins-config-volume\") pod \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\" (UID: \"e0d7662a-ed5f-4ee6-ae71-2e448222a4bf\") " Sep 10 17:33:11.853454 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.853430 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0d7662a-ed5f-4ee6-ae71-2e448222a4bf" (UID: "e0d7662a-ed5f-4ee6-ae71-2e448222a4bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:33:11.855737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.855686 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-kube-api-access-2mptl" (OuterVolumeSpecName: "kube-api-access-2mptl") pod "e0d7662a-ed5f-4ee6-ae71-2e448222a4bf" (UID: "e0d7662a-ed5f-4ee6-ae71-2e448222a4bf"). InnerVolumeSpecName "kube-api-access-2mptl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:11.860059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.860024 2557 scope.go:117] "RemoveContainer" containerID="50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132" Sep 10 17:33:11.860540 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:33:11.860518 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132\": container with ID starting with 50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132 not found: ID does not exist" containerID="50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132" Sep 10 17:33:11.860631 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.860550 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132"} err="failed to get container status \"50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132\": rpc error: code = NotFound desc = could not find container \"50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132\": container with ID starting with 50357c2c054c653f457934e5d5101fa3db84282b6f76b00f59b590ef884b0132 not found: ID does not exist" Sep 10 17:33:11.954477 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.954441 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mptl\" (UniqueName: \"kubernetes.io/projected/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-kube-api-access-2mptl\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:11.954477 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:11.954472 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:12.168827 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:12.168799 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767"] Sep 10 17:33:12.174586 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:12.174553 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-851d1193-57d6cdc89c-fj767"] Sep 10 17:33:13.387435 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:13.387403 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d7662a-ed5f-4ee6-ae71-2e448222a4bf" path="/var/lib/kubelet/pods/e0d7662a-ed5f-4ee6-ae71-2e448222a4bf/volumes" Sep 10 17:33:16.702650 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:16.702610 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl"] Sep 10 17:33:16.703135 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:16.702834 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" podUID="64635bf2-6766-4267-9851-f4a4f53a8bfb" containerName="istio-proxy" containerID="cri-o://5dc6f7a4f4fe184b4aad7e809bbe10477aec01b2ccf9b6b467c721d4abcb7a63" gracePeriod=30 Sep 10 17:33:18.304777 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:18.304741 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br"] Sep 10 17:33:18.305201 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:18.305015 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" containerID="cri-o://37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69" gracePeriod=30 Sep 10 17:33:19.622561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:19.622514 2557 patch_prober.go:28] interesting pod/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 17:33:19.622928 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:19.622588 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" podUID="64635bf2-6766-4267-9851-f4a4f53a8bfb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 17:33:21.360123 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:21.360094 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:33:21.364577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:21.364554 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:33:21.902538 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:21.902508 2557 generic.go:356] "Generic (PLEG): container finished" podID="64635bf2-6766-4267-9851-f4a4f53a8bfb" containerID="5dc6f7a4f4fe184b4aad7e809bbe10477aec01b2ccf9b6b467c721d4abcb7a63" exitCode=0 Sep 10 17:33:21.902680 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:21.902587 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" event={"ID":"64635bf2-6766-4267-9851-f4a4f53a8bfb","Type":"ContainerDied","Data":"5dc6f7a4f4fe184b4aad7e809bbe10477aec01b2ccf9b6b467c721d4abcb7a63"} Sep 10 17:33:21.956642 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:21.956617 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:33:22.051490 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051457 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-credential-socket\") pod \"64635bf2-6766-4267-9851-f4a4f53a8bfb\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " Sep 10 17:33:22.051652 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051497 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-certs\") pod \"64635bf2-6766-4267-9851-f4a4f53a8bfb\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " Sep 10 17:33:22.051652 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051528 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvg99\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-kube-api-access-jvg99\") pod \"64635bf2-6766-4267-9851-f4a4f53a8bfb\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " Sep 10 17:33:22.051652 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051605 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-socket\") pod \"64635bf2-6766-4267-9851-f4a4f53a8bfb\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " Sep 10 17:33:22.051652 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051628 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-token\") pod \"64635bf2-6766-4267-9851-f4a4f53a8bfb\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " Sep 10 17:33:22.051843 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051691 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-envoy\") pod \"64635bf2-6766-4267-9851-f4a4f53a8bfb\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " Sep 10 17:33:22.051843 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051722 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-data\") pod \"64635bf2-6766-4267-9851-f4a4f53a8bfb\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " Sep 10 17:33:22.051843 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051744 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-podinfo\") pod \"64635bf2-6766-4267-9851-f4a4f53a8bfb\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " Sep 10 17:33:22.051843 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051821 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64635bf2-6766-4267-9851-f4a4f53a8bfb-istiod-ca-cert\") pod \"64635bf2-6766-4267-9851-f4a4f53a8bfb\" (UID: \"64635bf2-6766-4267-9851-f4a4f53a8bfb\") " Sep 10 17:33:22.051843 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.051822 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64635bf2-6766-4267-9851-f4a4f53a8bfb" (UID: "64635bf2-6766-4267-9851-f4a4f53a8bfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:22.052122 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.052009 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-data" (OuterVolumeSpecName: "istio-data") pod "64635bf2-6766-4267-9851-f4a4f53a8bfb" (UID: "64635bf2-6766-4267-9851-f4a4f53a8bfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:22.052188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.052160 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64635bf2-6766-4267-9851-f4a4f53a8bfb" (UID: "64635bf2-6766-4267-9851-f4a4f53a8bfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:22.052337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.052246 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64635bf2-6766-4267-9851-f4a4f53a8bfb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64635bf2-6766-4267-9851-f4a4f53a8bfb" (UID: "64635bf2-6766-4267-9851-f4a4f53a8bfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:33:22.052337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.052307 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64635bf2-6766-4267-9851-f4a4f53a8bfb" (UID: "64635bf2-6766-4267-9851-f4a4f53a8bfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:22.052474 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.052364 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:22.052474 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.052379 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:22.052474 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.052392 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64635bf2-6766-4267-9851-f4a4f53a8bfb-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:22.052474 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.052404 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:22.053876 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.053845 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64635bf2-6766-4267-9851-f4a4f53a8bfb" (UID: "64635bf2-6766-4267-9851-f4a4f53a8bfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:22.053876 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.053867 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64635bf2-6766-4267-9851-f4a4f53a8bfb" (UID: "64635bf2-6766-4267-9851-f4a4f53a8bfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:33:22.054179 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.054161 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-kube-api-access-jvg99" (OuterVolumeSpecName: "kube-api-access-jvg99") pod "64635bf2-6766-4267-9851-f4a4f53a8bfb" (UID: "64635bf2-6766-4267-9851-f4a4f53a8bfb"). InnerVolumeSpecName "kube-api-access-jvg99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:22.054248 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.054180 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "64635bf2-6766-4267-9851-f4a4f53a8bfb" (UID: "64635bf2-6766-4267-9851-f4a4f53a8bfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:22.153013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.152965 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvg99\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-kube-api-access-jvg99\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:22.153013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.153007 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:22.153013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.153018 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:22.153013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.153026 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64635bf2-6766-4267-9851-f4a4f53a8bfb-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:22.153013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.153055 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64635bf2-6766-4267-9851-f4a4f53a8bfb-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:22.908737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.908701 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" Sep 10 17:33:22.909222 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.908736 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl" event={"ID":"64635bf2-6766-4267-9851-f4a4f53a8bfb","Type":"ContainerDied","Data":"52b79f76b32d5b00abc5efcac2456e8da34e8d45a039e1015b3abfb03ddf6c0a"} Sep 10 17:33:22.909222 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.908797 2557 scope.go:117] "RemoveContainer" containerID="5dc6f7a4f4fe184b4aad7e809bbe10477aec01b2ccf9b6b467c721d4abcb7a63" Sep 10 17:33:22.930800 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.930771 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl"] Sep 10 17:33:22.934485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:22.934463 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-8bd101-openshift-default-7745f7cff6-l6tgl"] Sep 10 17:33:23.386002 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.385966 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64635bf2-6766-4267-9851-f4a4f53a8bfb" path="/var/lib/kubelet/pods/64635bf2-6766-4267-9851-f4a4f53a8bfb/volumes" Sep 10 17:33:23.566097 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.566074 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:33:23.667495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667414 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-podinfo\") pod \"132308f6-43b1-48ea-8de1-effceb49f3f0\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " Sep 10 17:33:23.667495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667451 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-credential-socket\") pod \"132308f6-43b1-48ea-8de1-effceb49f3f0\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " Sep 10 17:33:23.667495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667469 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132308f6-43b1-48ea-8de1-effceb49f3f0-istiod-ca-cert\") pod \"132308f6-43b1-48ea-8de1-effceb49f3f0\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " Sep 10 17:33:23.667495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667488 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-certs\") pod \"132308f6-43b1-48ea-8de1-effceb49f3f0\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " Sep 10 17:33:23.667811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667534 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hct9\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-kube-api-access-7hct9\") pod \"132308f6-43b1-48ea-8de1-effceb49f3f0\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " Sep 10 17:33:23.667811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667587 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-socket\") pod \"132308f6-43b1-48ea-8de1-effceb49f3f0\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " Sep 10 17:33:23.667811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667610 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-data\") pod \"132308f6-43b1-48ea-8de1-effceb49f3f0\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " Sep 10 17:33:23.667811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667672 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-token\") pod \"132308f6-43b1-48ea-8de1-effceb49f3f0\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " Sep 10 17:33:23.667811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667695 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-envoy\") pod \"132308f6-43b1-48ea-8de1-effceb49f3f0\" (UID: \"132308f6-43b1-48ea-8de1-effceb49f3f0\") " Sep 10 17:33:23.667811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667763 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "132308f6-43b1-48ea-8de1-effceb49f3f0" (UID: "132308f6-43b1-48ea-8de1-effceb49f3f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:23.668151 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667882 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132308f6-43b1-48ea-8de1-effceb49f3f0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "132308f6-43b1-48ea-8de1-effceb49f3f0" (UID: "132308f6-43b1-48ea-8de1-effceb49f3f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:33:23.668151 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667981 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:23.668151 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667887 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "132308f6-43b1-48ea-8de1-effceb49f3f0" (UID: "132308f6-43b1-48ea-8de1-effceb49f3f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:23.668151 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.667956 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-data" (OuterVolumeSpecName: "istio-data") pod "132308f6-43b1-48ea-8de1-effceb49f3f0" (UID: "132308f6-43b1-48ea-8de1-effceb49f3f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:23.668420 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.668192 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "132308f6-43b1-48ea-8de1-effceb49f3f0" (UID: "132308f6-43b1-48ea-8de1-effceb49f3f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:23.669733 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.669710 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "132308f6-43b1-48ea-8de1-effceb49f3f0" (UID: "132308f6-43b1-48ea-8de1-effceb49f3f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:33:23.669883 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.669859 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "132308f6-43b1-48ea-8de1-effceb49f3f0" (UID: "132308f6-43b1-48ea-8de1-effceb49f3f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:23.669979 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.669958 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-kube-api-access-7hct9" (OuterVolumeSpecName: "kube-api-access-7hct9") pod "132308f6-43b1-48ea-8de1-effceb49f3f0" (UID: "132308f6-43b1-48ea-8de1-effceb49f3f0"). InnerVolumeSpecName "kube-api-access-7hct9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:23.670061 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.669965 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "132308f6-43b1-48ea-8de1-effceb49f3f0" (UID: "132308f6-43b1-48ea-8de1-effceb49f3f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:23.768923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.768890 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:23.768923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.768919 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132308f6-43b1-48ea-8de1-effceb49f3f0-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:23.768923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.768932 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:23.769144 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.768940 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hct9\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-kube-api-access-7hct9\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:23.769144 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.768949 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:23.769144 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.768957 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:23.769144 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.768965 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:23.769144 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.768972 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132308f6-43b1-48ea-8de1-effceb49f3f0-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:23.922278 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.922186 2557 generic.go:356] "Generic (PLEG): container finished" podID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerID="37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69" exitCode=0 Sep 10 17:33:23.922278 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.922244 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" Sep 10 17:33:23.922775 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.922247 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" event={"ID":"132308f6-43b1-48ea-8de1-effceb49f3f0","Type":"ContainerDied","Data":"37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69"} Sep 10 17:33:23.922775 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.922402 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br" event={"ID":"132308f6-43b1-48ea-8de1-effceb49f3f0","Type":"ContainerDied","Data":"94559c0708928f0d66d15e75f2721fa39dde87e058f831f478d98a0baa50f6d9"} Sep 10 17:33:23.922775 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.922436 2557 scope.go:117] "RemoveContainer" containerID="37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69" Sep 10 17:33:23.931626 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.931607 2557 scope.go:117] "RemoveContainer" containerID="1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310" Sep 10 17:33:23.941668 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.941623 2557 scope.go:117] "RemoveContainer" containerID="37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69" Sep 10 17:33:23.942017 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:33:23.941986 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69\": container with ID starting with 37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69 not found: ID does not exist" containerID="37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69" Sep 10 17:33:23.942138 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.942028 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69"} err="failed to get container status \"37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69\": rpc error: code = NotFound desc = could not find container \"37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69\": container with ID starting with 37e5a0a415c37b026e1a332ee0ac66559775ccbe2cf4facd379e3161d88d3b69 not found: ID does not exist" Sep 10 17:33:23.942138 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.942077 2557 scope.go:117] "RemoveContainer" containerID="1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310" Sep 10 17:33:23.942654 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:33:23.942620 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310\": container with ID starting with 1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310 not found: ID does not exist" containerID="1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310" Sep 10 17:33:23.942752 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.942650 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310"} err="failed to get container status \"1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310\": rpc error: code = NotFound desc = could not find container \"1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310\": container with ID starting with 1adaab59c0ddebc7de4a6050a7a5b9964dde3bf54ed4fd14eafcc7b1bd9a6310 not found: ID does not exist" Sep 10 17:33:23.944393 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.944374 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br"] Sep 10 17:33:23.948099 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:23.948078 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-851d1193-openshift-default-7d9889fc68-jw7br"] Sep 10 17:33:25.387131 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:25.387097 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" path="/var/lib/kubelet/pods/132308f6-43b1-48ea-8de1-effceb49f3f0/volumes" Sep 10 17:33:28.496405 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.496369 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx"] Sep 10 17:33:28.496784 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.496617 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" podUID="caba8334-452f-4598-9110-764809181b0c" containerName="maas-api" containerID="cri-o://a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179" gracePeriod=30 Sep 10 17:33:28.576951 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.576917 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w"] Sep 10 17:33:28.577453 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.577200 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" podUID="9334b0a0-9346-4db0-bdbc-1318669a6b1a" containerName="payload-processing" containerID="cri-o://72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe" gracePeriod=30 Sep 10 17:33:28.830423 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.830395 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:33:28.870210 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.870175 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:33:28.918734 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.918704 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9334b0a0-9346-4db0-bdbc-1318669a6b1a-plugins-config-volume\") pod \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\" (UID: \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\") " Sep 10 17:33:28.918909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.918769 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caba8334-452f-4598-9110-764809181b0c-openshift-service-ca\") pod \"caba8334-452f-4598-9110-764809181b0c\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " Sep 10 17:33:28.918909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.918831 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-metrics-certs\") pod \"caba8334-452f-4598-9110-764809181b0c\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " Sep 10 17:33:28.918909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.918854 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtqr8\" (UniqueName: \"kubernetes.io/projected/9334b0a0-9346-4db0-bdbc-1318669a6b1a-kube-api-access-gtqr8\") pod \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\" (UID: \"9334b0a0-9346-4db0-bdbc-1318669a6b1a\") " Sep 10 17:33:28.918909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.918894 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xlj2\" (UniqueName: \"kubernetes.io/projected/caba8334-452f-4598-9110-764809181b0c-kube-api-access-6xlj2\") pod \"caba8334-452f-4598-9110-764809181b0c\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " Sep 10 17:33:28.919161 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.918915 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-maas-api-tls\") pod \"caba8334-452f-4598-9110-764809181b0c\" (UID: \"caba8334-452f-4598-9110-764809181b0c\") " Sep 10 17:33:28.919161 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.919144 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9334b0a0-9346-4db0-bdbc-1318669a6b1a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9334b0a0-9346-4db0-bdbc-1318669a6b1a" (UID: "9334b0a0-9346-4db0-bdbc-1318669a6b1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:33:28.919257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.919183 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caba8334-452f-4598-9110-764809181b0c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "caba8334-452f-4598-9110-764809181b0c" (UID: "caba8334-452f-4598-9110-764809181b0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:33:28.921089 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.921061 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9334b0a0-9346-4db0-bdbc-1318669a6b1a-kube-api-access-gtqr8" (OuterVolumeSpecName: "kube-api-access-gtqr8") pod "9334b0a0-9346-4db0-bdbc-1318669a6b1a" (UID: "9334b0a0-9346-4db0-bdbc-1318669a6b1a"). InnerVolumeSpecName "kube-api-access-gtqr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:28.921202 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.921173 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "caba8334-452f-4598-9110-764809181b0c" (UID: "caba8334-452f-4598-9110-764809181b0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:33:28.921253 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.921231 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caba8334-452f-4598-9110-764809181b0c-kube-api-access-6xlj2" (OuterVolumeSpecName: "kube-api-access-6xlj2") pod "caba8334-452f-4598-9110-764809181b0c" (UID: "caba8334-452f-4598-9110-764809181b0c"). InnerVolumeSpecName "kube-api-access-6xlj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:28.921289 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.921261 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "caba8334-452f-4598-9110-764809181b0c" (UID: "caba8334-452f-4598-9110-764809181b0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:33:28.948720 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.948687 2557 generic.go:356] "Generic (PLEG): container finished" podID="caba8334-452f-4598-9110-764809181b0c" containerID="a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179" exitCode=0 Sep 10 17:33:28.948840 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.948728 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" Sep 10 17:33:28.948840 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.948775 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" event={"ID":"caba8334-452f-4598-9110-764809181b0c","Type":"ContainerDied","Data":"a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179"} Sep 10 17:33:28.948943 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.948837 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx" event={"ID":"caba8334-452f-4598-9110-764809181b0c","Type":"ContainerDied","Data":"a1ce8fb979d453b851c57afdd085adc91e23b52e6761134f93eccf1201e6fd93"} Sep 10 17:33:28.948943 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.948868 2557 scope.go:117] "RemoveContainer" containerID="a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179" Sep 10 17:33:28.950328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.950304 2557 generic.go:356] "Generic (PLEG): container finished" podID="9334b0a0-9346-4db0-bdbc-1318669a6b1a" containerID="72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe" exitCode=0 Sep 10 17:33:28.950459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.950345 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" Sep 10 17:33:28.950459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.950377 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" event={"ID":"9334b0a0-9346-4db0-bdbc-1318669a6b1a","Type":"ContainerDied","Data":"72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe"} Sep 10 17:33:28.950459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.950413 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w" event={"ID":"9334b0a0-9346-4db0-bdbc-1318669a6b1a","Type":"ContainerDied","Data":"b1f071003b8e10d24cce4580c45127fbe1886db54e7642652c69d7746b8e9fef"} Sep 10 17:33:28.967909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.967880 2557 scope.go:117] "RemoveContainer" containerID="a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179" Sep 10 17:33:28.968325 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:33:28.968295 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179\": container with ID starting with a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179 not found: ID does not exist" containerID="a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179" Sep 10 17:33:28.968408 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.968330 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179"} err="failed to get container status \"a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179\": rpc error: code = NotFound desc = could not find container \"a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179\": container with ID starting with a2ffab7e1ea95c048d20f60c30f7c8b238ac821bb7927bb8a0a116535e267179 not found: ID does not exist" Sep 10 17:33:28.968408 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.968348 2557 scope.go:117] "RemoveContainer" containerID="72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe" Sep 10 17:33:28.977239 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.977212 2557 scope.go:117] "RemoveContainer" containerID="72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe" Sep 10 17:33:28.977574 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:33:28.977551 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe\": container with ID starting with 72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe not found: ID does not exist" containerID="72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe" Sep 10 17:33:28.977654 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.977584 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe"} err="failed to get container status \"72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe\": rpc error: code = NotFound desc = could not find container \"72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe\": container with ID starting with 72c48f17088aa97bcecd17a045a57035e3ceb317497b1e6a4648493c105e1bfe not found: ID does not exist" Sep 10 17:33:28.978779 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.978752 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w"] Sep 10 17:33:28.981217 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.981199 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8c46e8-7fc4d9b9c7-sv29w"] Sep 10 17:33:28.990830 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.990793 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx"] Sep 10 17:33:28.995068 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:28.995025 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c46e8-66ff9f59-d4zxx"] Sep 10 17:33:29.020124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:29.020101 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caba8334-452f-4598-9110-764809181b0c-openshift-service-ca\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:29.020124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:29.020122 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-metrics-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:29.020253 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:29.020134 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtqr8\" (UniqueName: \"kubernetes.io/projected/9334b0a0-9346-4db0-bdbc-1318669a6b1a-kube-api-access-gtqr8\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:29.020253 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:29.020142 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlj2\" (UniqueName: \"kubernetes.io/projected/caba8334-452f-4598-9110-764809181b0c-kube-api-access-6xlj2\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:29.020253 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:29.020150 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caba8334-452f-4598-9110-764809181b0c-maas-api-tls\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:29.020253 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:29.020158 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9334b0a0-9346-4db0-bdbc-1318669a6b1a-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:29.386099 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:29.386060 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9334b0a0-9346-4db0-bdbc-1318669a6b1a" path="/var/lib/kubelet/pods/9334b0a0-9346-4db0-bdbc-1318669a6b1a/volumes" Sep 10 17:33:29.386504 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:29.386490 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caba8334-452f-4598-9110-764809181b0c" path="/var/lib/kubelet/pods/caba8334-452f-4598-9110-764809181b0c/volumes" Sep 10 17:33:32.797885 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.797851 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx"] Sep 10 17:33:32.798271 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798254 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caba8334-452f-4598-9110-764809181b0c" containerName="maas-api" Sep 10 17:33:32.798328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798274 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="caba8334-452f-4598-9110-764809181b0c" containerName="maas-api" Sep 10 17:33:32.798328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798288 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" Sep 10 17:33:32.798328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798294 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" Sep 10 17:33:32.798328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798303 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64635bf2-6766-4267-9851-f4a4f53a8bfb" containerName="istio-proxy" Sep 10 17:33:32.798328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798309 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="64635bf2-6766-4267-9851-f4a4f53a8bfb" containerName="istio-proxy" Sep 10 17:33:32.798328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798319 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d7662a-ed5f-4ee6-ae71-2e448222a4bf" containerName="payload-processing" Sep 10 17:33:32.798328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798324 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d7662a-ed5f-4ee6-ae71-2e448222a4bf" containerName="payload-processing" Sep 10 17:33:32.798539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798343 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9334b0a0-9346-4db0-bdbc-1318669a6b1a" containerName="payload-processing" Sep 10 17:33:32.798539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798350 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9334b0a0-9346-4db0-bdbc-1318669a6b1a" containerName="payload-processing" Sep 10 17:33:32.798539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798403 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9334b0a0-9346-4db0-bdbc-1318669a6b1a" containerName="payload-processing" Sep 10 17:33:32.798539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798414 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" Sep 10 17:33:32.798539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798423 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d7662a-ed5f-4ee6-ae71-2e448222a4bf" containerName="payload-processing" Sep 10 17:33:32.798539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798430 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="caba8334-452f-4598-9110-764809181b0c" containerName="maas-api" Sep 10 17:33:32.798539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798437 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="64635bf2-6766-4267-9851-f4a4f53a8bfb" containerName="istio-proxy" Sep 10 17:33:32.798539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.798443 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" Sep 10 17:33:32.801781 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.801762 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.803911 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.803889 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-52f72f-openshift-default-dockercfg-zxmgv\"" Sep 10 17:33:32.816183 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.816149 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx"] Sep 10 17:33:32.852804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.852770 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-data\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.852804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.852804 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-token\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.853004 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.852836 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-credential-socket\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.853004 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.852882 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istiod-ca-cert\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.853004 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.852958 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-certs\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.853004 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.852978 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjnk\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-kube-api-access-gjjnk\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.853188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.853069 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-socket\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.853188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.853087 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-envoy\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.853188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.853136 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-podinfo\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954014 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-certs\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954232 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954067 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjnk\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-kube-api-access-gjjnk\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954320 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954248 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-socket\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954320 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-envoy\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954386 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954348 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-podinfo\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954432 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-data\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954484 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954436 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-certs\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954484 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954445 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-token\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954578 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954514 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-credential-socket\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954578 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954529 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-socket\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954578 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954543 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istiod-ca-cert\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954777 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954757 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-data\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.954844 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.954773 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-credential-socket\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.955192 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.955174 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istiod-ca-cert\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.956451 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.956431 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-envoy\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.956820 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.956799 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-podinfo\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.961442 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.961420 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjnk\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-kube-api-access-gjjnk\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:32.961806 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:32.961779 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-token\") pod \"e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:33.112614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:33.112530 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:33.404590 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:33.404566 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx"] Sep 10 17:33:33.406420 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:33:33.406383 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf313f8aa_348e_41eb_9c0a_0f2dc18ab1b1.slice/crio-3766b87d7209dccf193c744e7f7ecd8f111c58514659468616e2321444f32a61 WatchSource:0}: Error finding container 3766b87d7209dccf193c744e7f7ecd8f111c58514659468616e2321444f32a61: Status 404 returned error can't find the container with id 3766b87d7209dccf193c744e7f7ecd8f111c58514659468616e2321444f32a61 Sep 10 17:33:33.408550 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:33.408518 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:33:33.408638 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:33.408582 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:33:33.408638 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:33.408613 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:33:33.983457 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:33.983415 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" event={"ID":"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1","Type":"ContainerStarted","Data":"5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10"} Sep 10 17:33:33.983918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:33.983504 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" event={"ID":"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1","Type":"ContainerStarted","Data":"3766b87d7209dccf193c744e7f7ecd8f111c58514659468616e2321444f32a61"} Sep 10 17:33:34.005006 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:34.004957 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" podStartSLOduration=2.004943283 podStartE2EDuration="2.004943283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:34.004143065 +0000 UTC m=+1213.102041427" watchObservedRunningTime="2026-09-10 17:33:34.004943283 +0000 UTC m=+1213.102841643" Sep 10 17:33:34.113449 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:34.113402 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:34.123090 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:34.123058 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:34.989391 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:34.989358 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:34.990400 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:34.990383 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:35.540778 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:35.540746 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2"] Sep 10 17:33:35.540991 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:35.540952 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" containerID="cri-o://9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b" gracePeriod=30 Sep 10 17:33:39.867849 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.867809 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p"] Sep 10 17:33:39.868304 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.868282 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" Sep 10 17:33:39.868304 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.868298 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="132308f6-43b1-48ea-8de1-effceb49f3f0" containerName="istio-proxy" Sep 10 17:33:39.870630 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.870613 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:39.872599 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.872573 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-52f72f\"" Sep 10 17:33:39.872599 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.872573 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-52f72f-dockercfg-62kkd\"" Sep 10 17:33:39.885219 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.885193 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p"] Sep 10 17:33:39.917371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.917328 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82074f68-b039-4f88-bc07-45f224b28f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p\" (UID: \"82074f68-b039-4f88-bc07-45f224b28f11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:39.917371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.917372 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjf8\" (UniqueName: \"kubernetes.io/projected/82074f68-b039-4f88-bc07-45f224b28f11-kube-api-access-7qjf8\") pod \"payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p\" (UID: \"82074f68-b039-4f88-bc07-45f224b28f11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:39.961802 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.961758 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94"] Sep 10 17:33:39.966170 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.966136 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:39.977798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:39.977088 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94"] Sep 10 17:33:40.018146 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.018112 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82074f68-b039-4f88-bc07-45f224b28f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p\" (UID: \"82074f68-b039-4f88-bc07-45f224b28f11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:40.018146 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.018147 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjf8\" (UniqueName: \"kubernetes.io/projected/82074f68-b039-4f88-bc07-45f224b28f11-kube-api-access-7qjf8\") pod \"payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p\" (UID: \"82074f68-b039-4f88-bc07-45f224b28f11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:40.018346 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.018176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2srv\" (UniqueName: \"kubernetes.io/projected/aa11b2fb-b949-4f5c-bafd-c50699aa050a-kube-api-access-s2srv\") pod \"payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94\" (UID: \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:40.018346 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.018207 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa11b2fb-b949-4f5c-bafd-c50699aa050a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94\" (UID: \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:40.018874 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.018854 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82074f68-b039-4f88-bc07-45f224b28f11-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p\" (UID: \"82074f68-b039-4f88-bc07-45f224b28f11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:40.027383 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.027342 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjf8\" (UniqueName: \"kubernetes.io/projected/82074f68-b039-4f88-bc07-45f224b28f11-kube-api-access-7qjf8\") pod \"payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p\" (UID: \"82074f68-b039-4f88-bc07-45f224b28f11\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:40.118672 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.118582 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa11b2fb-b949-4f5c-bafd-c50699aa050a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94\" (UID: \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:40.118852 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.118726 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2srv\" (UniqueName: \"kubernetes.io/projected/aa11b2fb-b949-4f5c-bafd-c50699aa050a-kube-api-access-s2srv\") pod \"payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94\" (UID: \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:40.119324 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.119299 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa11b2fb-b949-4f5c-bafd-c50699aa050a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94\" (UID: \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:40.126621 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.126590 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2srv\" (UniqueName: \"kubernetes.io/projected/aa11b2fb-b949-4f5c-bafd-c50699aa050a-kube-api-access-s2srv\") pod \"payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94\" (UID: \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:40.181811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.181768 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:40.279593 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.279559 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:40.454167 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.454139 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p"] Sep 10 17:33:40.456091 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:33:40.456062 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82074f68_b039_4f88_bc07_45f224b28f11.slice/crio-4e385fb011a73b93a247e93e666576da79d48f305a5cc69edb27b81c136e1d53 WatchSource:0}: Error finding container 4e385fb011a73b93a247e93e666576da79d48f305a5cc69edb27b81c136e1d53: Status 404 returned error can't find the container with id 4e385fb011a73b93a247e93e666576da79d48f305a5cc69edb27b81c136e1d53 Sep 10 17:33:40.555744 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.555716 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94"] Sep 10 17:33:40.610050 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:33:40.610001 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa11b2fb_b949_4f5c_bafd_c50699aa050a.slice/crio-f53518f4f47b8a8ed06cbc462127d9b3173de8fa840ae7115604500479a7156b WatchSource:0}: Error finding container f53518f4f47b8a8ed06cbc462127d9b3173de8fa840ae7115604500479a7156b: Status 404 returned error can't find the container with id f53518f4f47b8a8ed06cbc462127d9b3173de8fa840ae7115604500479a7156b Sep 10 17:33:40.794598 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.794574 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:33:40.826219 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826188 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7360973-3aae-4c11-956b-348bd684e3c0-istio-podinfo\") pod \"a7360973-3aae-4c11-956b-348bd684e3c0\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " Sep 10 17:33:40.826219 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826223 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-certs\") pod \"a7360973-3aae-4c11-956b-348bd684e3c0\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " Sep 10 17:33:40.826466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826275 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-credential-socket\") pod \"a7360973-3aae-4c11-956b-348bd684e3c0\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " Sep 10 17:33:40.826466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826313 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pskdn\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-kube-api-access-pskdn\") pod \"a7360973-3aae-4c11-956b-348bd684e3c0\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " Sep 10 17:33:40.826466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826346 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-envoy\") pod \"a7360973-3aae-4c11-956b-348bd684e3c0\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " Sep 10 17:33:40.826466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826376 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7360973-3aae-4c11-956b-348bd684e3c0-istiod-ca-cert\") pod \"a7360973-3aae-4c11-956b-348bd684e3c0\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " Sep 10 17:33:40.826466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826400 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-data\") pod \"a7360973-3aae-4c11-956b-348bd684e3c0\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " Sep 10 17:33:40.826466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826434 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-socket\") pod \"a7360973-3aae-4c11-956b-348bd684e3c0\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " Sep 10 17:33:40.826803 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826475 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-istio-token\") pod \"a7360973-3aae-4c11-956b-348bd684e3c0\" (UID: \"a7360973-3aae-4c11-956b-348bd684e3c0\") " Sep 10 17:33:40.826803 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826572 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7360973-3aae-4c11-956b-348bd684e3c0" (UID: "a7360973-3aae-4c11-956b-348bd684e3c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:40.826803 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826617 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7360973-3aae-4c11-956b-348bd684e3c0" (UID: "a7360973-3aae-4c11-956b-348bd684e3c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:40.826803 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826776 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:40.826803 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826794 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:40.827054 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826904 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7360973-3aae-4c11-956b-348bd684e3c0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7360973-3aae-4c11-956b-348bd684e3c0" (UID: "a7360973-3aae-4c11-956b-348bd684e3c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:33:40.827054 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.826910 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7360973-3aae-4c11-956b-348bd684e3c0" (UID: "a7360973-3aae-4c11-956b-348bd684e3c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:40.827187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.827157 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7360973-3aae-4c11-956b-348bd684e3c0" (UID: "a7360973-3aae-4c11-956b-348bd684e3c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:40.828777 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.828749 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7360973-3aae-4c11-956b-348bd684e3c0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7360973-3aae-4c11-956b-348bd684e3c0" (UID: "a7360973-3aae-4c11-956b-348bd684e3c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:33:40.829254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.829228 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-kube-api-access-pskdn" (OuterVolumeSpecName: "kube-api-access-pskdn") pod "a7360973-3aae-4c11-956b-348bd684e3c0" (UID: "a7360973-3aae-4c11-956b-348bd684e3c0"). InnerVolumeSpecName "kube-api-access-pskdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:40.829374 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.829228 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7360973-3aae-4c11-956b-348bd684e3c0" (UID: "a7360973-3aae-4c11-956b-348bd684e3c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:40.829374 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.829293 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7360973-3aae-4c11-956b-348bd684e3c0" (UID: "a7360973-3aae-4c11-956b-348bd684e3c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:33:40.927962 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.927932 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:40.927962 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.927959 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7360973-3aae-4c11-956b-348bd684e3c0-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:40.927962 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.927968 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pskdn\" (UniqueName: \"kubernetes.io/projected/a7360973-3aae-4c11-956b-348bd684e3c0-kube-api-access-pskdn\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:40.928409 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.927977 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:40.928409 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.927985 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7360973-3aae-4c11-956b-348bd684e3c0-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:40.928409 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.927993 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:40.928409 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:40.928000 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7360973-3aae-4c11-956b-348bd684e3c0-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:41.022720 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.022685 2557 generic.go:356] "Generic (PLEG): container finished" podID="a7360973-3aae-4c11-956b-348bd684e3c0" containerID="9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b" exitCode=0 Sep 10 17:33:41.022925 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.022743 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" Sep 10 17:33:41.022925 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.022770 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" event={"ID":"a7360973-3aae-4c11-956b-348bd684e3c0","Type":"ContainerDied","Data":"9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b"} Sep 10 17:33:41.022925 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.022838 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2" event={"ID":"a7360973-3aae-4c11-956b-348bd684e3c0","Type":"ContainerDied","Data":"2432a4aaa1c0b84c8667e354b38ae4154bbf88ec7ef9bed082b378a19f1047df"} Sep 10 17:33:41.022925 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.022870 2557 scope.go:117] "RemoveContainer" containerID="9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b" Sep 10 17:33:41.024863 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.024835 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" event={"ID":"aa11b2fb-b949-4f5c-bafd-c50699aa050a","Type":"ContainerStarted","Data":"32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7"} Sep 10 17:33:41.024988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.024896 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:41.024988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.024921 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" event={"ID":"aa11b2fb-b949-4f5c-bafd-c50699aa050a","Type":"ContainerStarted","Data":"f53518f4f47b8a8ed06cbc462127d9b3173de8fa840ae7115604500479a7156b"} Sep 10 17:33:41.027142 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.027116 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" event={"ID":"82074f68-b039-4f88-bc07-45f224b28f11","Type":"ContainerStarted","Data":"3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975"} Sep 10 17:33:41.027255 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.027168 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" event={"ID":"82074f68-b039-4f88-bc07-45f224b28f11","Type":"ContainerStarted","Data":"4e385fb011a73b93a247e93e666576da79d48f305a5cc69edb27b81c136e1d53"} Sep 10 17:33:41.027639 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.027619 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:41.045643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.043340 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" podStartSLOduration=2.043318919 podStartE2EDuration="2.043318919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:41.041758561 +0000 UTC m=+1220.139656932" watchObservedRunningTime="2026-09-10 17:33:41.043318919 +0000 UTC m=+1220.141217281" Sep 10 17:33:41.052105 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.052080 2557 scope.go:117] "RemoveContainer" containerID="e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b" Sep 10 17:33:41.060377 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.060326 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" podStartSLOduration=2.060310975 podStartE2EDuration="2.060310975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:41.058078401 +0000 UTC m=+1220.155976764" watchObservedRunningTime="2026-09-10 17:33:41.060310975 +0000 UTC m=+1220.158209391" Sep 10 17:33:41.061079 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.061057 2557 scope.go:117] "RemoveContainer" containerID="9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b" Sep 10 17:33:41.061407 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:33:41.061385 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b\": container with ID starting with 9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b not found: ID does not exist" containerID="9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b" Sep 10 17:33:41.061495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.061420 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b"} err="failed to get container status \"9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b\": rpc error: code = NotFound desc = could not find container \"9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b\": container with ID starting with 9905f11705947fa0837f99cce9e96d9018d68468b9241a867bfaf2c84d3ebe6b not found: ID does not exist" Sep 10 17:33:41.061495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.061441 2557 scope.go:117] "RemoveContainer" containerID="e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b" Sep 10 17:33:41.061741 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:33:41.061724 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b\": container with ID starting with e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b not found: ID does not exist" containerID="e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b" Sep 10 17:33:41.061782 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.061748 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b"} err="failed to get container status \"e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b\": rpc error: code = NotFound desc = could not find container \"e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b\": container with ID starting with e4244709ea93cdbd354a320954299e02202338e93719e038054d682dec7e901b not found: ID does not exist" Sep 10 17:33:41.074493 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.074462 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2"] Sep 10 17:33:41.077821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.077795 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8c46e8-openshift-default-66c9fb9c9d-rtdv2"] Sep 10 17:33:41.386851 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:41.386770 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" path="/var/lib/kubelet/pods/a7360973-3aae-4c11-956b-348bd684e3c0/volumes" Sep 10 17:33:47.334935 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.334900 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl"] Sep 10 17:33:47.335514 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.335132 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" podUID="20c3955d-45d3-4731-a9ed-77c2041c0380" containerName="payload-processing" containerID="cri-o://34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96" gracePeriod=30 Sep 10 17:33:47.377078 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.376270 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s"] Sep 10 17:33:47.377078 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.376529 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" podUID="fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd" containerName="payload-pre-processing" containerID="cri-o://a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b" gracePeriod=30 Sep 10 17:33:47.706009 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.705988 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:33:47.709616 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.709594 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:33:47.792297 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.792264 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgsr4\" (UniqueName: \"kubernetes.io/projected/20c3955d-45d3-4731-a9ed-77c2041c0380-kube-api-access-fgsr4\") pod \"20c3955d-45d3-4731-a9ed-77c2041c0380\" (UID: \"20c3955d-45d3-4731-a9ed-77c2041c0380\") " Sep 10 17:33:47.792461 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.792314 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-plugins-config-volume\") pod \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\" (UID: \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\") " Sep 10 17:33:47.792461 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.792352 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c3955d-45d3-4731-a9ed-77c2041c0380-plugins-config-volume\") pod \"20c3955d-45d3-4731-a9ed-77c2041c0380\" (UID: \"20c3955d-45d3-4731-a9ed-77c2041c0380\") " Sep 10 17:33:47.792461 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.792453 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmckx\" (UniqueName: \"kubernetes.io/projected/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-kube-api-access-qmckx\") pod \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\" (UID: \"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd\") " Sep 10 17:33:47.792649 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.792622 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd" (UID: "fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:33:47.792747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.792721 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c3955d-45d3-4731-a9ed-77c2041c0380-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20c3955d-45d3-4731-a9ed-77c2041c0380" (UID: "20c3955d-45d3-4731-a9ed-77c2041c0380"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:33:47.792808 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.792793 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:47.795024 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.794818 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-kube-api-access-qmckx" (OuterVolumeSpecName: "kube-api-access-qmckx") pod "fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd" (UID: "fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd"). InnerVolumeSpecName "kube-api-access-qmckx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:47.795180 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.795112 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c3955d-45d3-4731-a9ed-77c2041c0380-kube-api-access-fgsr4" (OuterVolumeSpecName: "kube-api-access-fgsr4") pod "20c3955d-45d3-4731-a9ed-77c2041c0380" (UID: "20c3955d-45d3-4731-a9ed-77c2041c0380"). InnerVolumeSpecName "kube-api-access-fgsr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:33:47.894124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.894026 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgsr4\" (UniqueName: \"kubernetes.io/projected/20c3955d-45d3-4731-a9ed-77c2041c0380-kube-api-access-fgsr4\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:47.894124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.894070 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c3955d-45d3-4731-a9ed-77c2041c0380-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:47.894124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:47.894080 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmckx\" (UniqueName: \"kubernetes.io/projected/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd-kube-api-access-qmckx\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:33:48.074791 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.074745 2557 generic.go:356] "Generic (PLEG): container finished" podID="20c3955d-45d3-4731-a9ed-77c2041c0380" containerID="34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96" exitCode=0 Sep 10 17:33:48.074966 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.074851 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" event={"ID":"20c3955d-45d3-4731-a9ed-77c2041c0380","Type":"ContainerDied","Data":"34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96"} Sep 10 17:33:48.074966 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.074907 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" event={"ID":"20c3955d-45d3-4731-a9ed-77c2041c0380","Type":"ContainerDied","Data":"8d41ab5059c46741d45ab75e05d2dc3eccda6ac15a9ce2684a55d203b5ae7545"} Sep 10 17:33:48.074966 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.074939 2557 scope.go:117] "RemoveContainer" containerID="34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96" Sep 10 17:33:48.075162 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.075145 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl" Sep 10 17:33:48.077248 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.077210 2557 generic.go:356] "Generic (PLEG): container finished" podID="fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd" containerID="a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b" exitCode=0 Sep 10 17:33:48.077627 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.077423 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" event={"ID":"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd","Type":"ContainerDied","Data":"a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b"} Sep 10 17:33:48.077627 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.077587 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" event={"ID":"fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd","Type":"ContainerDied","Data":"7b5f88ca38c172c7fada81ef7d9b8d4a42ee3a9f9edfaed923fa4aafae2a1250"} Sep 10 17:33:48.077627 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.077447 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s" Sep 10 17:33:48.086677 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.086508 2557 scope.go:117] "RemoveContainer" containerID="34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96" Sep 10 17:33:48.086838 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:33:48.086816 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96\": container with ID starting with 34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96 not found: ID does not exist" containerID="34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96" Sep 10 17:33:48.086909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.086846 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96"} err="failed to get container status \"34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96\": rpc error: code = NotFound desc = could not find container \"34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96\": container with ID starting with 34b4ead0037db8faa248935c6619b418117269f2f94a9d7b65bf772434e3fb96 not found: ID does not exist" Sep 10 17:33:48.086909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.086863 2557 scope.go:117] "RemoveContainer" containerID="a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b" Sep 10 17:33:48.094835 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.094818 2557 scope.go:117] "RemoveContainer" containerID="a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b" Sep 10 17:33:48.095085 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:33:48.095068 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b\": container with ID starting with a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b not found: ID does not exist" containerID="a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b" Sep 10 17:33:48.095139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.095091 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b"} err="failed to get container status \"a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b\": rpc error: code = NotFound desc = could not find container \"a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b\": container with ID starting with a1a89f82410f9ea50ded1a6844347dce8f681d57af7f99fef5623badab6b751b not found: ID does not exist" Sep 10 17:33:48.110635 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.110610 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl"] Sep 10 17:33:48.126057 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.126008 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-15ece2-6c59d7bd77-sbpcl"] Sep 10 17:33:48.151309 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.151215 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s"] Sep 10 17:33:48.163609 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:48.163568 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-15ece2-6bfdf4c774-pxb7s"] Sep 10 17:33:49.387341 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:49.387299 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c3955d-45d3-4731-a9ed-77c2041c0380" path="/var/lib/kubelet/pods/20c3955d-45d3-4731-a9ed-77c2041c0380/volumes" Sep 10 17:33:49.387836 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:49.387815 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd" path="/var/lib/kubelet/pods/fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd/volumes" Sep 10 17:33:51.031194 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:51.031148 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 17:33:51.031572 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:51.031219 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 17:33:52.036090 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:52.036066 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:33:53.041066 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:53.041013 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:33:53.114275 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:53.114241 2557 generic.go:356] "Generic (PLEG): container finished" podID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerID="5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10" exitCode=0 Sep 10 17:33:53.114476 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:53.114314 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" event={"ID":"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1","Type":"ContainerDied","Data":"5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10"} Sep 10 17:33:53.114731 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:53.114713 2557 scope.go:117] "RemoveContainer" containerID="5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10" Sep 10 17:33:53.115960 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:53.115926 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:33:53.116065 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:53.116003 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:33:53.116116 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:53.116061 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:33:54.121417 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:54.121386 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" event={"ID":"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1","Type":"ContainerStarted","Data":"6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77"} Sep 10 17:33:55.112712 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.112672 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:33:55.112950 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.112928 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 17:33:55.113064 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.112993 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.663508 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx"] Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.664085 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.664102 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.664153 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd" containerName="payload-pre-processing" Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.664163 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd" containerName="payload-pre-processing" Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.664175 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c3955d-45d3-4731-a9ed-77c2041c0380" containerName="payload-processing" Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.664184 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c3955d-45d3-4731-a9ed-77c2041c0380" containerName="payload-processing" Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.664261 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1d8fa2-251d-45ed-bcbb-3a6ff24042dd" containerName="payload-pre-processing" Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.664277 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" Sep 10 17:33:55.665059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.664290 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c3955d-45d3-4731-a9ed-77c2041c0380" containerName="payload-processing" Sep 10 17:33:55.666739 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.666720 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.675783 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.675755 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-3c30a2-openshift-default-dockercfg-m9zgs\"" Sep 10 17:33:55.687459 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.687429 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx"] Sep 10 17:33:55.778569 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.778530 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-podinfo\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.778755 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.778597 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istiod-ca-cert\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.778755 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.778626 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-credential-socket\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.778755 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.778642 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4fjl\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-kube-api-access-w4fjl\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.778755 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.778701 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-data\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.778755 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.778735 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-envoy\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.778959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.778799 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-certs\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.778959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.778837 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-token\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.778959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.778889 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-socket\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.879518 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879477 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-podinfo\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.879713 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879533 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istiod-ca-cert\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.879713 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879551 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-credential-socket\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.879713 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879669 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4fjl\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-kube-api-access-w4fjl\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.879924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879720 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-data\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.879924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879757 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-envoy\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.879924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879795 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-certs\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.879924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879826 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-token\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.879924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879902 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-socket\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.880152 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.879923 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-credential-socket\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.880290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.880265 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-data\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.880360 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.880336 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-socket\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.880407 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.880355 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istiod-ca-cert\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.880407 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.880394 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-certs\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.882094 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.882067 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-envoy\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.882218 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.882199 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-podinfo\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.889462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.889433 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-token\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.889792 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.889765 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4fjl\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-kube-api-access-w4fjl\") pod \"e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:55.989123 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:55.989015 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:56.114018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:56.113526 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 17:33:56.114018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:56.113589 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 17:33:56.270621 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:56.270574 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx"] Sep 10 17:33:56.280156 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:33:56.280116 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eefff4e_5d29_4f2c_af9a_5c1a1ec04150.slice/crio-12d68dc121630922d8354ea25f456a7b5ccd36d2f22b64343b78f865e2484a00 WatchSource:0}: Error finding container 12d68dc121630922d8354ea25f456a7b5ccd36d2f22b64343b78f865e2484a00: Status 404 returned error can't find the container with id 12d68dc121630922d8354ea25f456a7b5ccd36d2f22b64343b78f865e2484a00 Sep 10 17:33:56.282541 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:56.282504 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:33:56.282648 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:56.282590 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:33:56.282648 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:56.282635 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:33:57.113197 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:57.113158 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 17:33:57.113581 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:57.113218 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 17:33:57.140510 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:57.140477 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" event={"ID":"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150","Type":"ContainerStarted","Data":"01888e6cd46718260d2242c0a66c185d43d26d652451ae5a43fb630d9dea942d"} Sep 10 17:33:57.140671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:57.140526 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" event={"ID":"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150","Type":"ContainerStarted","Data":"12d68dc121630922d8354ea25f456a7b5ccd36d2f22b64343b78f865e2484a00"} Sep 10 17:33:57.165672 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:57.165608 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" podStartSLOduration=2.165586508 podStartE2EDuration="2.165586508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:33:57.164298692 +0000 UTC m=+1236.262197054" watchObservedRunningTime="2026-09-10 17:33:57.165586508 +0000 UTC m=+1236.263484869" Sep 10 17:33:57.989476 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:57.989439 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:57.994545 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:57.994517 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:58.113285 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:58.113251 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 17:33:58.113729 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:58.113313 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 17:33:58.146922 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:58.146874 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:58.148079 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:58.148051 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:33:59.113178 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:59.113145 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 10 17:33:59.113333 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:33:59.113209 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 10 17:34:00.117323 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:00.117289 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:34:00.117806 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:00.117584 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:34:00.118248 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:00.118228 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:34:02.351919 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.351880 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4"] Sep 10 17:34:02.352348 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.352263 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" Sep 10 17:34:02.352348 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.352275 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" Sep 10 17:34:02.352348 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.352349 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7360973-3aae-4c11-956b-348bd684e3c0" containerName="istio-proxy" Sep 10 17:34:02.354253 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.354237 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:02.361439 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.361110 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5912a3-dockercfg-qvg9k\"" Sep 10 17:34:02.367654 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.367622 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5912a3\"" Sep 10 17:34:02.379695 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.379665 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4"] Sep 10 17:34:02.442826 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.442791 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8rk\" (UniqueName: \"kubernetes.io/projected/b2608bac-782a-4b49-9256-464de2d889dd-kube-api-access-wf8rk\") pod \"payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4\" (UID: \"b2608bac-782a-4b49-9256-464de2d889dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:02.443018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.442875 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2608bac-782a-4b49-9256-464de2d889dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4\" (UID: \"b2608bac-782a-4b49-9256-464de2d889dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:02.543524 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.543474 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8rk\" (UniqueName: \"kubernetes.io/projected/b2608bac-782a-4b49-9256-464de2d889dd-kube-api-access-wf8rk\") pod \"payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4\" (UID: \"b2608bac-782a-4b49-9256-464de2d889dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:02.543719 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.543559 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2608bac-782a-4b49-9256-464de2d889dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4\" (UID: \"b2608bac-782a-4b49-9256-464de2d889dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:02.544381 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.544357 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2608bac-782a-4b49-9256-464de2d889dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4\" (UID: \"b2608bac-782a-4b49-9256-464de2d889dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:02.558308 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.558281 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8rk\" (UniqueName: \"kubernetes.io/projected/b2608bac-782a-4b49-9256-464de2d889dd-kube-api-access-wf8rk\") pod \"payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4\" (UID: \"b2608bac-782a-4b49-9256-464de2d889dd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:02.575340 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.575315 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54"] Sep 10 17:34:02.578065 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.578048 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:02.644639 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.644531 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00359170-6139-4167-a6ce-d76b3afef2e3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54\" (UID: \"00359170-6139-4167-a6ce-d76b3afef2e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:02.644639 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.644588 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdgl\" (UniqueName: \"kubernetes.io/projected/00359170-6139-4167-a6ce-d76b3afef2e3-kube-api-access-8zdgl\") pod \"payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54\" (UID: \"00359170-6139-4167-a6ce-d76b3afef2e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:02.663433 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.663392 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54"] Sep 10 17:34:02.670878 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.670841 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:02.745895 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.745850 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00359170-6139-4167-a6ce-d76b3afef2e3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54\" (UID: \"00359170-6139-4167-a6ce-d76b3afef2e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:02.746080 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.746002 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdgl\" (UniqueName: \"kubernetes.io/projected/00359170-6139-4167-a6ce-d76b3afef2e3-kube-api-access-8zdgl\") pod \"payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54\" (UID: \"00359170-6139-4167-a6ce-d76b3afef2e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:02.746848 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.746814 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00359170-6139-4167-a6ce-d76b3afef2e3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54\" (UID: \"00359170-6139-4167-a6ce-d76b3afef2e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:02.759437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.759407 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdgl\" (UniqueName: \"kubernetes.io/projected/00359170-6139-4167-a6ce-d76b3afef2e3-kube-api-access-8zdgl\") pod \"payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54\" (UID: \"00359170-6139-4167-a6ce-d76b3afef2e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:02.888925 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:02.888887 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:03.006869 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:03.006763 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4"] Sep 10 17:34:03.009995 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:34:03.009969 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2608bac_782a_4b49_9256_464de2d889dd.slice/crio-c32a6a84e82dd6f2825f54c763df3bd2083cac1dca53a060642bacbc0f02fd40 WatchSource:0}: Error finding container c32a6a84e82dd6f2825f54c763df3bd2083cac1dca53a060642bacbc0f02fd40: Status 404 returned error can't find the container with id c32a6a84e82dd6f2825f54c763df3bd2083cac1dca53a060642bacbc0f02fd40 Sep 10 17:34:03.174974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:03.174867 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" event={"ID":"b2608bac-782a-4b49-9256-464de2d889dd","Type":"ContainerStarted","Data":"55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c"} Sep 10 17:34:03.174974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:03.174941 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" event={"ID":"b2608bac-782a-4b49-9256-464de2d889dd","Type":"ContainerStarted","Data":"c32a6a84e82dd6f2825f54c763df3bd2083cac1dca53a060642bacbc0f02fd40"} Sep 10 17:34:03.174974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:03.174973 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:03.235246 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:03.235193 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" podStartSLOduration=1.23517566 podStartE2EDuration="1.23517566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:03.234183278 +0000 UTC m=+1242.332081640" watchObservedRunningTime="2026-09-10 17:34:03.23517566 +0000 UTC m=+1242.333074020" Sep 10 17:34:03.248284 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:03.248258 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54"] Sep 10 17:34:03.250659 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:34:03.250630 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00359170_6139_4167_a6ce_d76b3afef2e3.slice/crio-4be04119a05bc0f4dd7886a5cb6b40df5a172800c2cf0f9d7d27334a072ee3b7 WatchSource:0}: Error finding container 4be04119a05bc0f4dd7886a5cb6b40df5a172800c2cf0f9d7d27334a072ee3b7: Status 404 returned error can't find the container with id 4be04119a05bc0f4dd7886a5cb6b40df5a172800c2cf0f9d7d27334a072ee3b7 Sep 10 17:34:04.181805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:04.181757 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" event={"ID":"00359170-6139-4167-a6ce-d76b3afef2e3","Type":"ContainerStarted","Data":"d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b"} Sep 10 17:34:04.182281 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:04.181821 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" event={"ID":"00359170-6139-4167-a6ce-d76b3afef2e3","Type":"ContainerStarted","Data":"4be04119a05bc0f4dd7886a5cb6b40df5a172800c2cf0f9d7d27334a072ee3b7"} Sep 10 17:34:04.182281 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:04.181846 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:04.217723 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:04.217667 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" podStartSLOduration=2.217650881 podStartE2EDuration="2.217650881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:04.217249826 +0000 UTC m=+1243.315148190" watchObservedRunningTime="2026-09-10 17:34:04.217650881 +0000 UTC m=+1243.315549242" Sep 10 17:34:14.184462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:14.184384 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:34:15.189156 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:15.189124 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:34:21.277188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:21.277147 2557 generic.go:356] "Generic (PLEG): container finished" podID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerID="01888e6cd46718260d2242c0a66c185d43d26d652451ae5a43fb630d9dea942d" exitCode=0 Sep 10 17:34:21.277594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:21.277216 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" event={"ID":"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150","Type":"ContainerDied","Data":"01888e6cd46718260d2242c0a66c185d43d26d652451ae5a43fb630d9dea942d"} Sep 10 17:34:21.277657 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:21.277640 2557 scope.go:117] "RemoveContainer" containerID="01888e6cd46718260d2242c0a66c185d43d26d652451ae5a43fb630d9dea942d" Sep 10 17:34:21.278805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:21.278771 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:34:21.278903 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:21.278885 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:34:21.278953 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:21.278937 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:34:22.284100 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:22.284059 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" event={"ID":"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150","Type":"ContainerStarted","Data":"65f37d1b50cfe6aa8baa30f218207d24a1497c2a6bc19b35999a93191853c966"} Sep 10 17:34:22.989461 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:22.989419 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:34:22.989733 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:22.989710 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 17:34:22.989796 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:22.989774 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 17:34:23.990127 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:23.990089 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 17:34:23.990566 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:23.990144 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 17:34:24.989480 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:24.989437 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 17:34:24.989691 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:24.989508 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 17:34:25.989425 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:25.989389 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 17:34:25.989821 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:25.989442 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 17:34:26.989383 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:26.989345 2557 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Sep 10 17:34:26.989587 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:26.989421 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Sep 10 17:34:27.994047 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:27.994003 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:34:27.994487 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:27.994381 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:34:27.994984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:27.994965 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:34:34.840073 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.839978 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb"] Sep 10 17:34:34.843536 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.843509 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:34.847316 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.847296 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-68cae2e9-openshift-default-dockercfg-h8v7n\"" Sep 10 17:34:34.879274 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.879250 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb"] Sep 10 17:34:34.961659 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.961620 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-credential-socket\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:34.961867 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.961668 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-data\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:34.961867 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.961704 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-certs\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:34.961867 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.961786 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-podinfo\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:34.962076 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.961892 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-socket\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:34.962076 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.961926 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-token\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:34.962076 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.961950 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7x58\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-kube-api-access-m7x58\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:34.962076 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.961982 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-envoy\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:34.962076 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:34.962065 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istiod-ca-cert\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.062738 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.062699 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-socket\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.062738 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.062740 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-token\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.062984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.062756 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7x58\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-kube-api-access-m7x58\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.062984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.062777 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-envoy\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.062984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.062805 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istiod-ca-cert\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.062984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.062866 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-credential-socket\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.062984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.062887 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-data\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.062984 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.062918 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-certs\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.063312 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.062978 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-podinfo\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.063312 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.063290 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-socket\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.063422 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.063324 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-data\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.063422 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.063375 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-certs\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.063528 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.063505 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-credential-socket\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.063804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.063786 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istiod-ca-cert\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.065324 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.065302 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-envoy\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.065404 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.065373 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-podinfo\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.075773 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.075747 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7x58\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-kube-api-access-m7x58\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.078149 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.078127 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-token\") pod \"e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.154384 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.154301 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:35.463538 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.463513 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb"] Sep 10 17:34:35.465148 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:34:35.465118 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa1a37f_1714_4b67_97cc_dda2651d1ff5.slice/crio-3221392b319dac0ede3c3390b4a5b3036bde81934da5f749a64afef1961f78c3 WatchSource:0}: Error finding container 3221392b319dac0ede3c3390b4a5b3036bde81934da5f749a64afef1961f78c3: Status 404 returned error can't find the container with id 3221392b319dac0ede3c3390b4a5b3036bde81934da5f749a64afef1961f78c3 Sep 10 17:34:35.467354 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.467323 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:34:35.467467 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.467386 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:34:35.467467 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:35.467413 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:34:36.354284 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:36.354249 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" event={"ID":"1aa1a37f-1714-4b67-97cc-dda2651d1ff5","Type":"ContainerStarted","Data":"d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398"} Sep 10 17:34:36.354779 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:36.354298 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" event={"ID":"1aa1a37f-1714-4b67-97cc-dda2651d1ff5","Type":"ContainerStarted","Data":"3221392b319dac0ede3c3390b4a5b3036bde81934da5f749a64afef1961f78c3"} Sep 10 17:34:36.381616 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:36.381565 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" podStartSLOduration=2.381548453 podStartE2EDuration="2.381548453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:36.380676349 +0000 UTC m=+1275.478574713" watchObservedRunningTime="2026-09-10 17:34:36.381548453 +0000 UTC m=+1275.479446807" Sep 10 17:34:37.154585 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.154548 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:37.159670 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.159646 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:37.359283 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.359248 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:37.360091 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.360071 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:34:37.936135 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.936102 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp"] Sep 10 17:34:37.938468 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.938453 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:37.941993 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.941970 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c1fb61-dockercfg-jpzvs\"" Sep 10 17:34:37.942244 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.942227 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c1fb61\"" Sep 10 17:34:37.955596 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.955575 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp"] Sep 10 17:34:37.993880 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.993848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnmd\" (UniqueName: \"kubernetes.io/projected/4d6b97eb-c464-4c62-865a-064f5bb9428d-kube-api-access-lxnmd\") pod \"payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp\" (UID: \"4d6b97eb-c464-4c62-865a-064f5bb9428d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:37.994062 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:37.993937 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6b97eb-c464-4c62-865a-064f5bb9428d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp\" (UID: \"4d6b97eb-c464-4c62-865a-064f5bb9428d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:38.095334 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:38.095295 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnmd\" (UniqueName: \"kubernetes.io/projected/4d6b97eb-c464-4c62-865a-064f5bb9428d-kube-api-access-lxnmd\") pod \"payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp\" (UID: \"4d6b97eb-c464-4c62-865a-064f5bb9428d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:38.095518 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:38.095438 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6b97eb-c464-4c62-865a-064f5bb9428d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp\" (UID: \"4d6b97eb-c464-4c62-865a-064f5bb9428d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:38.096127 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:38.096109 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6b97eb-c464-4c62-865a-064f5bb9428d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp\" (UID: \"4d6b97eb-c464-4c62-865a-064f5bb9428d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:38.104958 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:38.104924 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnmd\" (UniqueName: \"kubernetes.io/projected/4d6b97eb-c464-4c62-865a-064f5bb9428d-kube-api-access-lxnmd\") pod \"payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp\" (UID: \"4d6b97eb-c464-4c62-865a-064f5bb9428d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:38.250163 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:38.250129 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:38.544548 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:38.544520 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp"] Sep 10 17:34:38.546535 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:34:38.546504 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6b97eb_c464_4c62_865a_064f5bb9428d.slice/crio-a69915f412ff0d12e49170680c8ebfb16552961fd02f1398aeb767c53606d15e WatchSource:0}: Error finding container a69915f412ff0d12e49170680c8ebfb16552961fd02f1398aeb767c53606d15e: Status 404 returned error can't find the container with id a69915f412ff0d12e49170680c8ebfb16552961fd02f1398aeb767c53606d15e Sep 10 17:34:39.375672 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:39.375626 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" event={"ID":"4d6b97eb-c464-4c62-865a-064f5bb9428d","Type":"ContainerStarted","Data":"9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd"} Sep 10 17:34:39.375862 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:39.375689 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" event={"ID":"4d6b97eb-c464-4c62-865a-064f5bb9428d","Type":"ContainerStarted","Data":"a69915f412ff0d12e49170680c8ebfb16552961fd02f1398aeb767c53606d15e"} Sep 10 17:34:39.375862 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:39.375739 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:39.413952 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:39.413890 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" podStartSLOduration=2.413874572 podStartE2EDuration="2.413874572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:39.412581504 +0000 UTC m=+1278.510479866" watchObservedRunningTime="2026-09-10 17:34:39.413874572 +0000 UTC m=+1278.511772972" Sep 10 17:34:44.301234 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.301199 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt"] Sep 10 17:34:44.304002 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.303986 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:34:44.306420 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.306399 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-68cae2e9-dockercfg-54x6n\"" Sep 10 17:34:44.306499 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.306406 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-68cae2e9\"" Sep 10 17:34:44.330218 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.330191 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt"] Sep 10 17:34:44.459540 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.459494 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd1c571-ca1c-4468-8a49-e40af82320c2-plugins-config-volume\") pod \"payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt\" (UID: \"dfd1c571-ca1c-4468-8a49-e40af82320c2\") " pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:34:44.459887 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.459864 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kljd\" (UniqueName: \"kubernetes.io/projected/dfd1c571-ca1c-4468-8a49-e40af82320c2-kube-api-access-6kljd\") pod \"payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt\" (UID: \"dfd1c571-ca1c-4468-8a49-e40af82320c2\") " pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:34:44.560737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.560648 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd1c571-ca1c-4468-8a49-e40af82320c2-plugins-config-volume\") pod \"payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt\" (UID: \"dfd1c571-ca1c-4468-8a49-e40af82320c2\") " pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:34:44.560737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.560713 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kljd\" (UniqueName: \"kubernetes.io/projected/dfd1c571-ca1c-4468-8a49-e40af82320c2-kube-api-access-6kljd\") pod \"payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt\" (UID: \"dfd1c571-ca1c-4468-8a49-e40af82320c2\") " pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:34:44.561437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.561416 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd1c571-ca1c-4468-8a49-e40af82320c2-plugins-config-volume\") pod \"payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt\" (UID: \"dfd1c571-ca1c-4468-8a49-e40af82320c2\") " pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:34:44.572276 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.572253 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kljd\" (UniqueName: \"kubernetes.io/projected/dfd1c571-ca1c-4468-8a49-e40af82320c2-kube-api-access-6kljd\") pod \"payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt\" (UID: \"dfd1c571-ca1c-4468-8a49-e40af82320c2\") " pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:34:44.617013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.616971 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:34:44.897066 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:44.896979 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt"] Sep 10 17:34:44.899494 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:34:44.899470 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd1c571_ca1c_4468_8a49_e40af82320c2.slice/crio-3366ce4f01ba8fb0a3e6a7bfebb5563b502f319add74d4dff9a0429a2db22fc2 WatchSource:0}: Error finding container 3366ce4f01ba8fb0a3e6a7bfebb5563b502f319add74d4dff9a0429a2db22fc2: Status 404 returned error can't find the container with id 3366ce4f01ba8fb0a3e6a7bfebb5563b502f319add74d4dff9a0429a2db22fc2 Sep 10 17:34:45.408880 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:45.408844 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" event={"ID":"dfd1c571-ca1c-4468-8a49-e40af82320c2","Type":"ContainerStarted","Data":"3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6"} Sep 10 17:34:45.409275 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:45.408934 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" event={"ID":"dfd1c571-ca1c-4468-8a49-e40af82320c2","Type":"ContainerStarted","Data":"3366ce4f01ba8fb0a3e6a7bfebb5563b502f319add74d4dff9a0429a2db22fc2"} Sep 10 17:34:45.409275 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:45.408966 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:34:45.434157 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:45.434101 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" podStartSLOduration=1.4340844879999999 podStartE2EDuration="1.434084488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:45.430943418 +0000 UTC m=+1284.528841780" watchObservedRunningTime="2026-09-10 17:34:45.434084488 +0000 UTC m=+1284.531982849" Sep 10 17:34:51.389172 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:51.389143 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:34:56.416109 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:34:56.416078 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:35:10.537221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:10.537188 2557 generic.go:356] "Generic (PLEG): container finished" podID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerID="d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398" exitCode=0 Sep 10 17:35:10.537662 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:10.537254 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" event={"ID":"1aa1a37f-1714-4b67-97cc-dda2651d1ff5","Type":"ContainerDied","Data":"d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398"} Sep 10 17:35:10.537662 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:10.537629 2557 scope.go:117] "RemoveContainer" containerID="d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398" Sep 10 17:35:10.544857 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:10.544815 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:35:10.544963 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:10.544910 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:35:10.544963 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:10.544952 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:35:11.545267 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:11.545222 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" event={"ID":"1aa1a37f-1714-4b67-97cc-dda2651d1ff5","Type":"ContainerStarted","Data":"2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221"} Sep 10 17:35:12.155367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:12.155321 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:35:12.155811 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:12.155779 2557 patch_prober.go:28] interesting pod/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:35:12.155930 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:12.155841 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 17:35:13.155440 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:13.155394 2557 patch_prober.go:28] interesting pod/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:35:13.155914 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:13.155464 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 17:35:14.156121 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:14.155619 2557 patch_prober.go:28] interesting pod/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:35:14.156121 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:14.155702 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 17:35:15.155478 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:15.155434 2557 patch_prober.go:28] interesting pod/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:35:15.155693 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:15.155506 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 17:35:16.155480 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:16.155437 2557 patch_prober.go:28] interesting pod/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 10 17:35:16.155971 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:16.155522 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 10 17:35:17.158817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:17.158783 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:35:17.159293 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:17.159109 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:35:17.159659 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:35:17.159637 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:36:18.881084 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:18.881050 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt"] Sep 10 17:36:18.881585 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:18.881312 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" podUID="dfd1c571-ca1c-4468-8a49-e40af82320c2" containerName="payload-processing" containerID="cri-o://3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6" gracePeriod=30 Sep 10 17:36:19.144540 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.144466 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:36:19.245279 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.245242 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kljd\" (UniqueName: \"kubernetes.io/projected/dfd1c571-ca1c-4468-8a49-e40af82320c2-kube-api-access-6kljd\") pod \"dfd1c571-ca1c-4468-8a49-e40af82320c2\" (UID: \"dfd1c571-ca1c-4468-8a49-e40af82320c2\") " Sep 10 17:36:19.245456 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.245341 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd1c571-ca1c-4468-8a49-e40af82320c2-plugins-config-volume\") pod \"dfd1c571-ca1c-4468-8a49-e40af82320c2\" (UID: \"dfd1c571-ca1c-4468-8a49-e40af82320c2\") " Sep 10 17:36:19.246009 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.245978 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd1c571-ca1c-4468-8a49-e40af82320c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfd1c571-ca1c-4468-8a49-e40af82320c2" (UID: "dfd1c571-ca1c-4468-8a49-e40af82320c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:36:19.247960 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.247854 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd1c571-ca1c-4468-8a49-e40af82320c2-kube-api-access-6kljd" (OuterVolumeSpecName: "kube-api-access-6kljd") pod "dfd1c571-ca1c-4468-8a49-e40af82320c2" (UID: "dfd1c571-ca1c-4468-8a49-e40af82320c2"). InnerVolumeSpecName "kube-api-access-6kljd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:36:19.346879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.346838 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kljd\" (UniqueName: \"kubernetes.io/projected/dfd1c571-ca1c-4468-8a49-e40af82320c2-kube-api-access-6kljd\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:36:19.346879 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.346880 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd1c571-ca1c-4468-8a49-e40af82320c2-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:36:19.897606 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.897569 2557 generic.go:356] "Generic (PLEG): container finished" podID="dfd1c571-ca1c-4468-8a49-e40af82320c2" containerID="3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6" exitCode=0 Sep 10 17:36:19.898153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.897614 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" event={"ID":"dfd1c571-ca1c-4468-8a49-e40af82320c2","Type":"ContainerDied","Data":"3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6"} Sep 10 17:36:19.898153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.897657 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" event={"ID":"dfd1c571-ca1c-4468-8a49-e40af82320c2","Type":"ContainerDied","Data":"3366ce4f01ba8fb0a3e6a7bfebb5563b502f319add74d4dff9a0429a2db22fc2"} Sep 10 17:36:19.898153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.897680 2557 scope.go:117] "RemoveContainer" containerID="3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6" Sep 10 17:36:19.898153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.897616 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt" Sep 10 17:36:19.906569 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.906541 2557 scope.go:117] "RemoveContainer" containerID="3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6" Sep 10 17:36:19.906843 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:36:19.906823 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6\": container with ID starting with 3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6 not found: ID does not exist" containerID="3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6" Sep 10 17:36:19.906899 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.906851 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6"} err="failed to get container status \"3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6\": rpc error: code = NotFound desc = could not find container \"3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6\": container with ID starting with 3a48338cd927abf162b32256f463d889501db0f2755bdcc68429ac814a08a9c6 not found: ID does not exist" Sep 10 17:36:19.915798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.915775 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt"] Sep 10 17:36:19.919483 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:19.919461 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-68cae2e9-fd856dcfc-pnbdt"] Sep 10 17:36:21.386447 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:36:21.386413 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfd1c571-ca1c-4468-8a49-e40af82320c2" path="/var/lib/kubelet/pods/dfd1c571-ca1c-4468-8a49-e40af82320c2/volumes" Sep 10 17:37:10.252843 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:10.252811 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb"] Sep 10 17:37:10.253341 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:10.253079 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" containerID="cri-o://2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221" gracePeriod=30 Sep 10 17:37:15.303241 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:37:15.303209 2557 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa1a37f_1714_4b67_97cc_dda2651d1ff5.slice/crio-conmon-2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221.scope\": RecentStats: unable to find data in memory cache]" Sep 10 17:37:15.512346 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.512322 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:37:15.612900 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.612833 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-token\") pod \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " Sep 10 17:37:15.612900 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.612863 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7x58\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-kube-api-access-m7x58\") pod \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " Sep 10 17:37:15.613119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.612918 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-data\") pod \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " Sep 10 17:37:15.613119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.612938 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-credential-socket\") pod \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " Sep 10 17:37:15.613119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.612959 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-socket\") pod \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " Sep 10 17:37:15.613119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.612985 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istiod-ca-cert\") pod \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " Sep 10 17:37:15.613119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.613029 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-envoy\") pod \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " Sep 10 17:37:15.613119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.613081 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-certs\") pod \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " Sep 10 17:37:15.613440 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.613126 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-podinfo\") pod \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\" (UID: \"1aa1a37f-1714-4b67-97cc-dda2651d1ff5\") " Sep 10 17:37:15.613440 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.613212 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-data" (OuterVolumeSpecName: "istio-data") pod "1aa1a37f-1714-4b67-97cc-dda2651d1ff5" (UID: "1aa1a37f-1714-4b67-97cc-dda2651d1ff5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:37:15.613549 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.613460 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:37:15.613549 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.613472 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1aa1a37f-1714-4b67-97cc-dda2651d1ff5" (UID: "1aa1a37f-1714-4b67-97cc-dda2651d1ff5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:37:15.613687 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.613668 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1aa1a37f-1714-4b67-97cc-dda2651d1ff5" (UID: "1aa1a37f-1714-4b67-97cc-dda2651d1ff5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:37:15.613788 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.613756 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1aa1a37f-1714-4b67-97cc-dda2651d1ff5" (UID: "1aa1a37f-1714-4b67-97cc-dda2651d1ff5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:37:15.613834 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.613596 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1aa1a37f-1714-4b67-97cc-dda2651d1ff5" (UID: "1aa1a37f-1714-4b67-97cc-dda2651d1ff5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:37:15.615192 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.615168 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1aa1a37f-1714-4b67-97cc-dda2651d1ff5" (UID: "1aa1a37f-1714-4b67-97cc-dda2651d1ff5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:37:15.615296 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.615187 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-token" (OuterVolumeSpecName: "istio-token") pod "1aa1a37f-1714-4b67-97cc-dda2651d1ff5" (UID: "1aa1a37f-1714-4b67-97cc-dda2651d1ff5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:37:15.615473 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.615451 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1aa1a37f-1714-4b67-97cc-dda2651d1ff5" (UID: "1aa1a37f-1714-4b67-97cc-dda2651d1ff5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:37:15.615526 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.615496 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-kube-api-access-m7x58" (OuterVolumeSpecName: "kube-api-access-m7x58") pod "1aa1a37f-1714-4b67-97cc-dda2651d1ff5" (UID: "1aa1a37f-1714-4b67-97cc-dda2651d1ff5"). InnerVolumeSpecName "kube-api-access-m7x58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:37:15.714544 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.714522 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:37:15.714544 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.714543 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:37:15.714671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.714552 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:37:15.714671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.714561 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:37:15.714671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.714569 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:37:15.714671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.714576 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:37:15.714671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.714584 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:37:15.714671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:15.714591 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7x58\" (UniqueName: \"kubernetes.io/projected/1aa1a37f-1714-4b67-97cc-dda2651d1ff5-kube-api-access-m7x58\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:37:16.179065 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.179014 2557 generic.go:356] "Generic (PLEG): container finished" podID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerID="2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221" exitCode=0 Sep 10 17:37:16.179216 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.179082 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" Sep 10 17:37:16.179216 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.179087 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" event={"ID":"1aa1a37f-1714-4b67-97cc-dda2651d1ff5","Type":"ContainerDied","Data":"2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221"} Sep 10 17:37:16.179216 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.179140 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb" event={"ID":"1aa1a37f-1714-4b67-97cc-dda2651d1ff5","Type":"ContainerDied","Data":"3221392b319dac0ede3c3390b4a5b3036bde81934da5f749a64afef1961f78c3"} Sep 10 17:37:16.179216 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.179169 2557 scope.go:117] "RemoveContainer" containerID="2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221" Sep 10 17:37:16.188336 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.188317 2557 scope.go:117] "RemoveContainer" containerID="d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398" Sep 10 17:37:16.196227 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.196212 2557 scope.go:117] "RemoveContainer" containerID="2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221" Sep 10 17:37:16.196485 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:37:16.196466 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221\": container with ID starting with 2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221 not found: ID does not exist" containerID="2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221" Sep 10 17:37:16.196539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.196494 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221"} err="failed to get container status \"2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221\": rpc error: code = NotFound desc = could not find container \"2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221\": container with ID starting with 2da120888519da871895795b3fcc44f6d1a4d3012306878fbcb4827ac7648221 not found: ID does not exist" Sep 10 17:37:16.196539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.196510 2557 scope.go:117] "RemoveContainer" containerID="d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398" Sep 10 17:37:16.196743 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:37:16.196726 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398\": container with ID starting with d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398 not found: ID does not exist" containerID="d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398" Sep 10 17:37:16.196779 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.196749 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398"} err="failed to get container status \"d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398\": rpc error: code = NotFound desc = could not find container \"d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398\": container with ID starting with d479f4e322b4c1f97e02237f86d2518e48c891e7c34aef29ebb9856e991e4398 not found: ID does not exist" Sep 10 17:37:16.201128 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.201106 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb"] Sep 10 17:37:16.208296 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:16.208273 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-68cae2e9-openshift-default-858b68dbbd-dfgzb"] Sep 10 17:37:17.386442 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:37:17.386407 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" path="/var/lib/kubelet/pods/1aa1a37f-1714-4b67-97cc-dda2651d1ff5/volumes" Sep 10 17:38:05.086171 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.086131 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb"] Sep 10 17:38:05.086768 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.086748 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" Sep 10 17:38:05.086817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.086773 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" Sep 10 17:38:05.086817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.086790 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd1c571-ca1c-4468-8a49-e40af82320c2" containerName="payload-processing" Sep 10 17:38:05.086817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.086798 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd1c571-ca1c-4468-8a49-e40af82320c2" containerName="payload-processing" Sep 10 17:38:05.086817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.086812 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" Sep 10 17:38:05.086950 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.086822 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" Sep 10 17:38:05.086986 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.086948 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd1c571-ca1c-4468-8a49-e40af82320c2" containerName="payload-processing" Sep 10 17:38:05.086986 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.086962 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" Sep 10 17:38:05.090316 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.090300 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.092512 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.092490 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b5f818-openshift-default-dockercfg-895gj\"" Sep 10 17:38:05.104387 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.104366 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb"] Sep 10 17:38:05.121825 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.121796 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-certs\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.121959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.121827 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-data\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.121959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.121858 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75pzn\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-kube-api-access-75pzn\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.121959 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.121921 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-podinfo\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.122103 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.121972 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-socket\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.122103 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.121998 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-envoy\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.122103 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.122072 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-token\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.122103 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.122100 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-credential-socket\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.122237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.122124 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istiod-ca-cert\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.222570 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.222532 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-certs\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.222570 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.222567 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-data\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.222837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.222595 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75pzn\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-kube-api-access-75pzn\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.222837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.222614 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-podinfo\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.222837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.222644 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-socket\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.222837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.222676 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-envoy\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.222837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.222728 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-token\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.222837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.222758 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-credential-socket\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.222837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.222791 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istiod-ca-cert\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.223245 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.223004 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-certs\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.223245 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.223181 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-socket\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.223245 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.223211 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-data\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.223359 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.223260 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-credential-socket\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.223596 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.223574 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istiod-ca-cert\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.225046 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.225018 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-envoy\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.225178 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.225158 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-podinfo\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.230570 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.230547 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-token\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.230682 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.230583 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75pzn\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-kube-api-access-75pzn\") pod \"e2e-api-a-b5f818-openshift-default-65594659d8-wggpb\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.400818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.400739 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:05.676975 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.676947 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb"] Sep 10 17:38:05.678392 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:38:05.678365 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e11f413_5d01_4ec0_8d69_f39d4da9e6ec.slice/crio-e321084a91be388e1a0403121f68176a929f6d2e4d1dd53636cb2a6c2fe642d0 WatchSource:0}: Error finding container e321084a91be388e1a0403121f68176a929f6d2e4d1dd53636cb2a6c2fe642d0: Status 404 returned error can't find the container with id e321084a91be388e1a0403121f68176a929f6d2e4d1dd53636cb2a6c2fe642d0 Sep 10 17:38:05.680665 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.680632 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:38:05.680747 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.680697 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:38:05.680788 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:05.680746 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:38:06.430955 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:06.430914 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" event={"ID":"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec","Type":"ContainerStarted","Data":"4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2"} Sep 10 17:38:06.431347 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:06.430977 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" event={"ID":"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec","Type":"ContainerStarted","Data":"e321084a91be388e1a0403121f68176a929f6d2e4d1dd53636cb2a6c2fe642d0"} Sep 10 17:38:06.454525 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:06.454464 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" podStartSLOduration=1.454450003 podStartE2EDuration="1.454450003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:06.451895998 +0000 UTC m=+1485.549794358" watchObservedRunningTime="2026-09-10 17:38:06.454450003 +0000 UTC m=+1485.552348363" Sep 10 17:38:07.400857 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:07.400826 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:07.405706 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:07.405684 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:07.436385 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:07.436357 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:07.437240 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:07.437223 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:12.741206 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.741173 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s"] Sep 10 17:38:12.741640 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.741626 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa1a37f-1714-4b67-97cc-dda2651d1ff5" containerName="istio-proxy" Sep 10 17:38:12.744837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.744822 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:12.749474 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.749454 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b5f818-dockercfg-p9d4h\"" Sep 10 17:38:12.750021 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.750001 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b5f818\"" Sep 10 17:38:12.758740 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.758720 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s"] Sep 10 17:38:12.790762 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.790736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvkv\" (UniqueName: \"kubernetes.io/projected/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-kube-api-access-vlvkv\") pod \"payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s\" (UID: \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:12.790866 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.790844 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s\" (UID: \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:12.892385 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.892349 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvkv\" (UniqueName: \"kubernetes.io/projected/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-kube-api-access-vlvkv\") pod \"payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s\" (UID: \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:12.892541 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.892436 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s\" (UID: \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:12.893118 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.893100 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s\" (UID: \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:12.900926 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:12.900898 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvkv\" (UniqueName: \"kubernetes.io/projected/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-kube-api-access-vlvkv\") pod \"payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s\" (UID: \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\") " pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:13.055296 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:13.055207 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:13.334337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:13.334308 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s"] Sep 10 17:38:13.336354 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:38:13.336324 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f51e6b4_d07c_4c2a_82fb_d15f1cf3640f.slice/crio-a2032c30a7ca594fb7cb12bd1a5eb6464f2bf2df44e4c42d88b268ddc865cdcc WatchSource:0}: Error finding container a2032c30a7ca594fb7cb12bd1a5eb6464f2bf2df44e4c42d88b268ddc865cdcc: Status 404 returned error can't find the container with id a2032c30a7ca594fb7cb12bd1a5eb6464f2bf2df44e4c42d88b268ddc865cdcc Sep 10 17:38:13.469404 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:13.469373 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" event={"ID":"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f","Type":"ContainerStarted","Data":"95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a"} Sep 10 17:38:13.469561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:13.469424 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" event={"ID":"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f","Type":"ContainerStarted","Data":"a2032c30a7ca594fb7cb12bd1a5eb6464f2bf2df44e4c42d88b268ddc865cdcc"} Sep 10 17:38:13.469561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:13.469460 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:13.499098 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:13.499000 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" podStartSLOduration=1.498985844 podStartE2EDuration="1.498985844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:13.48927942 +0000 UTC m=+1492.587177781" watchObservedRunningTime="2026-09-10 17:38:13.498985844 +0000 UTC m=+1492.596884204" Sep 10 17:38:21.390076 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:21.390024 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:38:21.396562 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:21.396542 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:38:24.477192 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:24.477160 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:38:31.572375 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:31.572335 2557 generic.go:356] "Generic (PLEG): container finished" podID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerID="4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2" exitCode=0 Sep 10 17:38:31.572817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:31.572406 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" event={"ID":"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec","Type":"ContainerDied","Data":"4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2"} Sep 10 17:38:31.572817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:31.572799 2557 scope.go:117] "RemoveContainer" containerID="4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2" Sep 10 17:38:31.573909 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:31.573876 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:38:31.573990 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:31.573951 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:38:31.574028 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:31.574005 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:38:32.578951 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:32.578918 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" event={"ID":"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec","Type":"ContainerStarted","Data":"198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46"} Sep 10 17:38:33.401738 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:33.401701 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:33.403015 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:33.402988 2557 patch_prober.go:28] interesting pod/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 17:38:33.403149 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:33.403057 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 17:38:34.401429 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:34.401392 2557 patch_prober.go:28] interesting pod/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 17:38:34.401836 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:34.401451 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 17:38:35.401242 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:35.401209 2557 patch_prober.go:28] interesting pod/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 17:38:35.401410 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:35.401261 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 17:38:36.401224 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:36.401188 2557 patch_prober.go:28] interesting pod/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 17:38:36.401680 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:36.401244 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 17:38:37.401450 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:37.401418 2557 patch_prober.go:28] interesting pod/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 10 17:38:37.401830 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:37.401472 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 10 17:38:37.436468 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:37.436445 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:38.405077 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:38.405026 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:38.406297 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:38.406270 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:38:39.509377 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.509341 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx"] Sep 10 17:38:39.512311 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.512289 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.514387 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.514367 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a45272-openshift-default-dockercfg-bgbwg\"" Sep 10 17:38:39.523701 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.523680 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx"] Sep 10 17:38:39.638832 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.638800 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-data\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.639027 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.638849 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-credential-socket\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.639027 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.638944 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-token\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.639027 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.638994 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-envoy\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.639027 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.639025 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-certs\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.639220 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.639079 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-podinfo\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.639220 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.639110 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a4fc89-6e21-4c75-9916-b70da82202d9-istiod-ca-cert\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.639220 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.639151 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkkk\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-kube-api-access-qpkkk\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.639220 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.639176 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-socket\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.739763 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.739734 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-credential-socket\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.739939 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.739796 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-token\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.739939 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.739831 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-envoy\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.739939 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.739860 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-certs\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.739939 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.739877 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-podinfo\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.739939 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.739904 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a4fc89-6e21-4c75-9916-b70da82202d9-istiod-ca-cert\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.739939 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.739934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkkk\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-kube-api-access-qpkkk\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.740302 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.739979 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-socket\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.740302 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.740055 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-data\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.740483 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.740426 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-socket\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.740628 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.740595 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-certs\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.740786 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.740761 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-data\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.740884 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.740862 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-credential-socket\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.741188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.741169 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a4fc89-6e21-4c75-9916-b70da82202d9-istiod-ca-cert\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.742511 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.742487 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-envoy\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.742727 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.742708 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-podinfo\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.749268 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.749247 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-token\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.749554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.749536 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkkk\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-kube-api-access-qpkkk\") pod \"e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:39.822758 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:39.822677 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:40.104177 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:40.104147 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx"] Sep 10 17:38:40.105680 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:38:40.105650 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a4fc89_6e21_4c75_9916_b70da82202d9.slice/crio-638cb91b83e7c8bc8a978b2cb177d6eb12dc420143b68eb57fb08259222dc7e3 WatchSource:0}: Error finding container 638cb91b83e7c8bc8a978b2cb177d6eb12dc420143b68eb57fb08259222dc7e3: Status 404 returned error can't find the container with id 638cb91b83e7c8bc8a978b2cb177d6eb12dc420143b68eb57fb08259222dc7e3 Sep 10 17:38:40.107866 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:40.107835 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:38:40.107940 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:40.107900 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:38:40.107940 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:40.107928 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:38:40.623988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:40.623949 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" event={"ID":"84a4fc89-6e21-4c75-9916-b70da82202d9","Type":"ContainerStarted","Data":"530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8"} Sep 10 17:38:40.624379 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:40.624002 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" event={"ID":"84a4fc89-6e21-4c75-9916-b70da82202d9","Type":"ContainerStarted","Data":"638cb91b83e7c8bc8a978b2cb177d6eb12dc420143b68eb57fb08259222dc7e3"} Sep 10 17:38:40.647048 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:40.646821 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" podStartSLOduration=1.646801406 podStartE2EDuration="1.646801406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:40.643817865 +0000 UTC m=+1519.741716238" watchObservedRunningTime="2026-09-10 17:38:40.646801406 +0000 UTC m=+1519.744699767" Sep 10 17:38:40.823287 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:40.823248 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:41.828249 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:41.828219 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:42.634188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:42.634149 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:42.635161 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:42.635139 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:38:47.738567 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:47.738490 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs"] Sep 10 17:38:47.741433 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:47.741417 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:38:47.743351 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:47.743326 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a45272\"" Sep 10 17:38:47.743490 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:47.743393 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a45272-dockercfg-qt5pc\"" Sep 10 17:38:47.750804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:47.750778 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs"] Sep 10 17:38:47.923839 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:47.923799 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs\" (UID: \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:38:47.924010 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:47.923850 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ssj2\" (UniqueName: \"kubernetes.io/projected/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-kube-api-access-6ssj2\") pod \"payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs\" (UID: \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:38:48.025176 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.025139 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs\" (UID: \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:38:48.025368 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.025215 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssj2\" (UniqueName: \"kubernetes.io/projected/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-kube-api-access-6ssj2\") pod \"payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs\" (UID: \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:38:48.025923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.025892 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs\" (UID: \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:38:48.032485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.032458 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ssj2\" (UniqueName: \"kubernetes.io/projected/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-kube-api-access-6ssj2\") pod \"payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs\" (UID: \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:38:48.053349 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.053328 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:38:48.323168 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.323146 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs"] Sep 10 17:38:48.324960 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:38:48.324927 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11602a01_c1c5_4ce7_806c_7bc1ba1625c3.slice/crio-ef3f749e1cc44ff0ad9a109bd170ed37aa70ff9aff665a9e5d16f9430c7f032c WatchSource:0}: Error finding container ef3f749e1cc44ff0ad9a109bd170ed37aa70ff9aff665a9e5d16f9430c7f032c: Status 404 returned error can't find the container with id ef3f749e1cc44ff0ad9a109bd170ed37aa70ff9aff665a9e5d16f9430c7f032c Sep 10 17:38:48.671172 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.671081 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" event={"ID":"11602a01-c1c5-4ce7-806c-7bc1ba1625c3","Type":"ContainerStarted","Data":"ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019"} Sep 10 17:38:48.671172 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.671151 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:38:48.671380 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.671179 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" event={"ID":"11602a01-c1c5-4ce7-806c-7bc1ba1625c3","Type":"ContainerStarted","Data":"ef3f749e1cc44ff0ad9a109bd170ed37aa70ff9aff665a9e5d16f9430c7f032c"} Sep 10 17:38:48.688150 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:48.688094 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" podStartSLOduration=1.6880753689999999 podStartE2EDuration="1.688075369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:48.686729496 +0000 UTC m=+1527.784627860" watchObservedRunningTime="2026-09-10 17:38:48.688075369 +0000 UTC m=+1527.785973731" Sep 10 17:38:59.679101 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:38:59.679063 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:39:04.759796 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:04.759759 2557 generic.go:356] "Generic (PLEG): container finished" podID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerID="530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8" exitCode=0 Sep 10 17:39:04.760258 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:04.759803 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" event={"ID":"84a4fc89-6e21-4c75-9916-b70da82202d9","Type":"ContainerDied","Data":"530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8"} Sep 10 17:39:04.760258 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:04.760186 2557 scope.go:117] "RemoveContainer" containerID="530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8" Sep 10 17:39:04.761390 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:04.761357 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:39:04.761464 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:04.761431 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:39:04.761502 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:04.761463 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:39:05.771540 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:05.771495 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" event={"ID":"84a4fc89-6e21-4c75-9916-b70da82202d9","Type":"ContainerStarted","Data":"9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d"} Sep 10 17:39:05.822860 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:05.822830 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:39:05.823008 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:05.822991 2557 patch_prober.go:28] interesting pod/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 17:39:05.823089 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:05.823051 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 17:39:06.823976 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:06.823937 2557 patch_prober.go:28] interesting pod/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 17:39:06.824432 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:06.823995 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 17:39:07.823813 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:07.823780 2557 patch_prober.go:28] interesting pod/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 17:39:07.823995 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:07.823838 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 17:39:08.823646 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:08.823609 2557 patch_prober.go:28] interesting pod/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 17:39:08.823841 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:08.823672 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 17:39:09.823355 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:09.823319 2557 patch_prober.go:28] interesting pod/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 17:39:09.823766 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:09.823377 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 17:39:10.823253 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:10.823218 2557 patch_prober.go:28] interesting pod/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 10 17:39:10.823440 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:10.823278 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 10 17:39:11.827091 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:11.827062 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:39:11.827551 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:11.827340 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:39:11.828093 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:11.828074 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:39:31.529772 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.529728 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s"] Sep 10 17:39:31.530264 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.530053 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" podUID="0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f" containerName="payload-processing" containerID="cri-o://95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a" gracePeriod=30 Sep 10 17:39:31.798094 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.798064 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:39:31.909095 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.909059 2557 generic.go:356] "Generic (PLEG): container finished" podID="0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f" containerID="95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a" exitCode=0 Sep 10 17:39:31.909257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.909113 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" Sep 10 17:39:31.909257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.909136 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" event={"ID":"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f","Type":"ContainerDied","Data":"95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a"} Sep 10 17:39:31.909257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.909186 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s" event={"ID":"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f","Type":"ContainerDied","Data":"a2032c30a7ca594fb7cb12bd1a5eb6464f2bf2df44e4c42d88b268ddc865cdcc"} Sep 10 17:39:31.909257 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.909209 2557 scope.go:117] "RemoveContainer" containerID="95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a" Sep 10 17:39:31.918241 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.918225 2557 scope.go:117] "RemoveContainer" containerID="95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a" Sep 10 17:39:31.918519 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:39:31.918501 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a\": container with ID starting with 95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a not found: ID does not exist" containerID="95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a" Sep 10 17:39:31.918560 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.918528 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a"} err="failed to get container status \"95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a\": rpc error: code = NotFound desc = could not find container \"95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a\": container with ID starting with 95341bc5888458757b24624747c955ed5177275e18a38565a14cfbf696edda1a not found: ID does not exist" Sep 10 17:39:31.925980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.925954 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-plugins-config-volume\") pod \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\" (UID: \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\") " Sep 10 17:39:31.926145 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.926004 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlvkv\" (UniqueName: \"kubernetes.io/projected/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-kube-api-access-vlvkv\") pod \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\" (UID: \"0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f\") " Sep 10 17:39:31.926416 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.926390 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f" (UID: "0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:39:31.927974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:31.927950 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-kube-api-access-vlvkv" (OuterVolumeSpecName: "kube-api-access-vlvkv") pod "0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f" (UID: "0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f"). InnerVolumeSpecName "kube-api-access-vlvkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:39:32.027737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:32.027706 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:32.027990 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:32.027736 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlvkv\" (UniqueName: \"kubernetes.io/projected/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f-kube-api-access-vlvkv\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:32.230333 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:32.230305 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s"] Sep 10 17:39:32.234423 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:32.234386 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b5f818-69c4b49555-mb95s"] Sep 10 17:39:33.386145 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:33.386114 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f" path="/var/lib/kubelet/pods/0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f/volumes" Sep 10 17:39:39.205492 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:39.205456 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb"] Sep 10 17:39:39.206013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:39.205653 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" containerID="cri-o://198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46" gracePeriod=30 Sep 10 17:39:44.458856 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.458833 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:39:44.530317 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530289 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-credential-socket\") pod \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " Sep 10 17:39:44.530471 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530336 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istiod-ca-cert\") pod \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " Sep 10 17:39:44.530471 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530368 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-podinfo\") pod \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " Sep 10 17:39:44.530471 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530406 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-envoy\") pod \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " Sep 10 17:39:44.530647 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530549 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-data\") pod \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " Sep 10 17:39:44.530647 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530633 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75pzn\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-kube-api-access-75pzn\") pod \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " Sep 10 17:39:44.530762 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530659 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-token\") pod \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " Sep 10 17:39:44.530762 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530553 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" (UID: "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:39:44.530762 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530686 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-certs\") pod \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " Sep 10 17:39:44.530762 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530727 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-socket\") pod \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\" (UID: \"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec\") " Sep 10 17:39:44.530762 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530742 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" (UID: "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:39:44.531050 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530766 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" (UID: "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:39:44.531050 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.530938 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" (UID: "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:39:44.531174 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.531064 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:44.531174 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.531106 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:44.531174 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.531121 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:44.531174 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.531124 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" (UID: "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:39:44.531174 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.531135 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:44.532658 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.532633 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" (UID: "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:39:44.532779 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.532762 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" (UID: "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:39:44.532961 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.532937 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" (UID: "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:39:44.533080 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.532939 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-kube-api-access-75pzn" (OuterVolumeSpecName: "kube-api-access-75pzn") pod "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" (UID: "7e11f413-5d01-4ec0-8d69-f39d4da9e6ec"). InnerVolumeSpecName "kube-api-access-75pzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:39:44.631885 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.631864 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:44.631885 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.631886 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:44.632013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.631897 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75pzn\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-kube-api-access-75pzn\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:44.632013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.631906 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:44.632013 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.631914 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:44.978639 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.978556 2557 generic.go:356] "Generic (PLEG): container finished" podID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerID="198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46" exitCode=0 Sep 10 17:39:44.978639 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.978626 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" Sep 10 17:39:44.978836 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.978661 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" event={"ID":"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec","Type":"ContainerDied","Data":"198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46"} Sep 10 17:39:44.978836 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.978720 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb" event={"ID":"7e11f413-5d01-4ec0-8d69-f39d4da9e6ec","Type":"ContainerDied","Data":"e321084a91be388e1a0403121f68176a929f6d2e4d1dd53636cb2a6c2fe642d0"} Sep 10 17:39:44.978836 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.978744 2557 scope.go:117] "RemoveContainer" containerID="198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46" Sep 10 17:39:44.991512 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.991492 2557 scope.go:117] "RemoveContainer" containerID="4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2" Sep 10 17:39:44.999383 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.999340 2557 scope.go:117] "RemoveContainer" containerID="198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46" Sep 10 17:39:44.999666 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:39:44.999646 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46\": container with ID starting with 198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46 not found: ID does not exist" containerID="198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46" Sep 10 17:39:44.999739 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.999675 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46"} err="failed to get container status \"198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46\": rpc error: code = NotFound desc = could not find container \"198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46\": container with ID starting with 198b832e3cc3584f6f902361669b399c1a329d3c2c89e4ca7a0f427e2182da46 not found: ID does not exist" Sep 10 17:39:44.999739 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.999691 2557 scope.go:117] "RemoveContainer" containerID="4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2" Sep 10 17:39:44.999909 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:39:44.999895 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2\": container with ID starting with 4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2 not found: ID does not exist" containerID="4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2" Sep 10 17:39:44.999953 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:44.999911 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2"} err="failed to get container status \"4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2\": rpc error: code = NotFound desc = could not find container \"4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2\": container with ID starting with 4092b5b4e512ac2bc2302463e253a8cae11a8dd50c618ab669339184d754ecf2 not found: ID does not exist" Sep 10 17:39:45.004647 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:45.004625 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb"] Sep 10 17:39:45.009598 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:45.009576 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b5f818-openshift-default-65594659d8-wggpb"] Sep 10 17:39:45.387139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:45.387110 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" path="/var/lib/kubelet/pods/7e11f413-5d01-4ec0-8d69-f39d4da9e6ec/volumes" Sep 10 17:39:55.311420 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:55.311390 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs"] Sep 10 17:39:55.311867 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:55.311640 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" podUID="11602a01-c1c5-4ce7-806c-7bc1ba1625c3" containerName="payload-pre-processing" containerID="cri-o://ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019" gracePeriod=30 Sep 10 17:39:55.575647 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:55.575588 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:39:55.627989 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:55.627950 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-plugins-config-volume\") pod \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\" (UID: \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\") " Sep 10 17:39:55.628170 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:55.628012 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ssj2\" (UniqueName: \"kubernetes.io/projected/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-kube-api-access-6ssj2\") pod \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\" (UID: \"11602a01-c1c5-4ce7-806c-7bc1ba1625c3\") " Sep 10 17:39:55.628405 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:55.628380 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11602a01-c1c5-4ce7-806c-7bc1ba1625c3" (UID: "11602a01-c1c5-4ce7-806c-7bc1ba1625c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:39:55.630124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:55.630100 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-kube-api-access-6ssj2" (OuterVolumeSpecName: "kube-api-access-6ssj2") pod "11602a01-c1c5-4ce7-806c-7bc1ba1625c3" (UID: "11602a01-c1c5-4ce7-806c-7bc1ba1625c3"). InnerVolumeSpecName "kube-api-access-6ssj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:39:55.729603 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:55.729556 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ssj2\" (UniqueName: \"kubernetes.io/projected/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-kube-api-access-6ssj2\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:55.729603 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:55.729587 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11602a01-c1c5-4ce7-806c-7bc1ba1625c3-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:39:56.037641 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:56.037598 2557 generic.go:356] "Generic (PLEG): container finished" podID="11602a01-c1c5-4ce7-806c-7bc1ba1625c3" containerID="ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019" exitCode=0 Sep 10 17:39:56.037818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:56.037646 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" event={"ID":"11602a01-c1c5-4ce7-806c-7bc1ba1625c3","Type":"ContainerDied","Data":"ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019"} Sep 10 17:39:56.037818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:56.037699 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" event={"ID":"11602a01-c1c5-4ce7-806c-7bc1ba1625c3","Type":"ContainerDied","Data":"ef3f749e1cc44ff0ad9a109bd170ed37aa70ff9aff665a9e5d16f9430c7f032c"} Sep 10 17:39:56.037818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:56.037727 2557 scope.go:117] "RemoveContainer" containerID="ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019" Sep 10 17:39:56.037818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:56.037653 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs" Sep 10 17:39:56.047608 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:56.047578 2557 scope.go:117] "RemoveContainer" containerID="ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019" Sep 10 17:39:56.047921 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:39:56.047897 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019\": container with ID starting with ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019 not found: ID does not exist" containerID="ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019" Sep 10 17:39:56.048019 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:56.047926 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019"} err="failed to get container status \"ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019\": rpc error: code = NotFound desc = could not find container \"ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019\": container with ID starting with ea28d0f1d4944067f812c9d89f1d3bff4cdc38d8dddb36cda68872e7b4502019 not found: ID does not exist" Sep 10 17:39:56.069501 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:56.069476 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs"] Sep 10 17:39:56.085665 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:56.085645 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a45272-759d7d7f98-ms2zs"] Sep 10 17:39:57.386809 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:39:57.386780 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11602a01-c1c5-4ce7-806c-7bc1ba1625c3" path="/var/lib/kubelet/pods/11602a01-c1c5-4ce7-806c-7bc1ba1625c3/volumes" Sep 10 17:40:02.596983 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:02.596953 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx"] Sep 10 17:40:02.597401 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:02.597190 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" containerID="cri-o://9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d" gracePeriod=30 Sep 10 17:40:07.859337 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.859277 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:40:07.944363 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944335 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-data\") pod \"84a4fc89-6e21-4c75-9916-b70da82202d9\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " Sep 10 17:40:07.944363 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944368 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-credential-socket\") pod \"84a4fc89-6e21-4c75-9916-b70da82202d9\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " Sep 10 17:40:07.944571 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944390 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-token\") pod \"84a4fc89-6e21-4c75-9916-b70da82202d9\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " Sep 10 17:40:07.944571 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944414 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a4fc89-6e21-4c75-9916-b70da82202d9-istiod-ca-cert\") pod \"84a4fc89-6e21-4c75-9916-b70da82202d9\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " Sep 10 17:40:07.944571 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944511 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-certs\") pod \"84a4fc89-6e21-4c75-9916-b70da82202d9\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " Sep 10 17:40:07.944712 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944574 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpkkk\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-kube-api-access-qpkkk\") pod \"84a4fc89-6e21-4c75-9916-b70da82202d9\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " Sep 10 17:40:07.944712 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944606 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-socket\") pod \"84a4fc89-6e21-4c75-9916-b70da82202d9\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " Sep 10 17:40:07.944712 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944652 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-envoy\") pod \"84a4fc89-6e21-4c75-9916-b70da82202d9\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " Sep 10 17:40:07.944712 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944691 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-podinfo\") pod \"84a4fc89-6e21-4c75-9916-b70da82202d9\" (UID: \"84a4fc89-6e21-4c75-9916-b70da82202d9\") " Sep 10 17:40:07.944924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944644 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "84a4fc89-6e21-4c75-9916-b70da82202d9" (UID: "84a4fc89-6e21-4c75-9916-b70da82202d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:07.944924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944655 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "84a4fc89-6e21-4c75-9916-b70da82202d9" (UID: "84a4fc89-6e21-4c75-9916-b70da82202d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:07.944924 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944894 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "84a4fc89-6e21-4c75-9916-b70da82202d9" (UID: "84a4fc89-6e21-4c75-9916-b70da82202d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:07.945190 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.944939 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a4fc89-6e21-4c75-9916-b70da82202d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "84a4fc89-6e21-4c75-9916-b70da82202d9" (UID: "84a4fc89-6e21-4c75-9916-b70da82202d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:40:07.945190 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.945029 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:07.945190 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.945078 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:07.945190 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.945093 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:07.945190 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.945105 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a4fc89-6e21-4c75-9916-b70da82202d9-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:07.945190 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.945175 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "84a4fc89-6e21-4c75-9916-b70da82202d9" (UID: "84a4fc89-6e21-4c75-9916-b70da82202d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:07.946828 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.946800 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "84a4fc89-6e21-4c75-9916-b70da82202d9" (UID: "84a4fc89-6e21-4c75-9916-b70da82202d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:40:07.946931 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.946823 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-kube-api-access-qpkkk" (OuterVolumeSpecName: "kube-api-access-qpkkk") pod "84a4fc89-6e21-4c75-9916-b70da82202d9" (UID: "84a4fc89-6e21-4c75-9916-b70da82202d9"). InnerVolumeSpecName "kube-api-access-qpkkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:40:07.946931 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.946842 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "84a4fc89-6e21-4c75-9916-b70da82202d9" (UID: "84a4fc89-6e21-4c75-9916-b70da82202d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:07.946931 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:07.946909 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "84a4fc89-6e21-4c75-9916-b70da82202d9" (UID: "84a4fc89-6e21-4c75-9916-b70da82202d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:40:08.046517 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.046487 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:08.046517 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.046512 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:08.046517 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.046523 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpkkk\" (UniqueName: \"kubernetes.io/projected/84a4fc89-6e21-4c75-9916-b70da82202d9-kube-api-access-qpkkk\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:08.046721 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.046532 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:08.046721 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.046541 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a4fc89-6e21-4c75-9916-b70da82202d9-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:08.109923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.109847 2557 generic.go:356] "Generic (PLEG): container finished" podID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerID="9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d" exitCode=0 Sep 10 17:40:08.109923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.109900 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" Sep 10 17:40:08.110119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.109933 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" event={"ID":"84a4fc89-6e21-4c75-9916-b70da82202d9","Type":"ContainerDied","Data":"9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d"} Sep 10 17:40:08.110119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.109994 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx" event={"ID":"84a4fc89-6e21-4c75-9916-b70da82202d9","Type":"ContainerDied","Data":"638cb91b83e7c8bc8a978b2cb177d6eb12dc420143b68eb57fb08259222dc7e3"} Sep 10 17:40:08.110119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.110024 2557 scope.go:117] "RemoveContainer" containerID="9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d" Sep 10 17:40:08.119064 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.119045 2557 scope.go:117] "RemoveContainer" containerID="530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8" Sep 10 17:40:08.127312 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.127292 2557 scope.go:117] "RemoveContainer" containerID="9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d" Sep 10 17:40:08.127573 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:40:08.127555 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d\": container with ID starting with 9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d not found: ID does not exist" containerID="9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d" Sep 10 17:40:08.127622 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.127582 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d"} err="failed to get container status \"9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d\": rpc error: code = NotFound desc = could not find container \"9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d\": container with ID starting with 9c6f819ea912e48088e7136284c70697d7e4a6f5ef609045854911e2bf069f5d not found: ID does not exist" Sep 10 17:40:08.127622 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.127599 2557 scope.go:117] "RemoveContainer" containerID="530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8" Sep 10 17:40:08.127841 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:40:08.127825 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8\": container with ID starting with 530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8 not found: ID does not exist" containerID="530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8" Sep 10 17:40:08.127881 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.127846 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8"} err="failed to get container status \"530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8\": rpc error: code = NotFound desc = could not find container \"530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8\": container with ID starting with 530973e28d41b4876cd0fc6848e78527ee0ebb50a33db639c5bf69082ff0e0c8 not found: ID does not exist" Sep 10 17:40:08.132189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.132151 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx"] Sep 10 17:40:08.135296 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:08.135268 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a45272-openshift-default-f98bf5d56-fssfx"] Sep 10 17:40:09.386156 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:09.386127 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" path="/var/lib/kubelet/pods/84a4fc89-6e21-4c75-9916-b70da82202d9/volumes" Sep 10 17:40:13.574244 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.574212 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94"] Sep 10 17:40:13.574798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.574510 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" podUID="aa11b2fb-b949-4f5c-bafd-c50699aa050a" containerName="payload-processing" containerID="cri-o://32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7" gracePeriod=30 Sep 10 17:40:13.583539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.583515 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p"] Sep 10 17:40:13.583723 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.583704 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" podUID="82074f68-b039-4f88-bc07-45f224b28f11" containerName="payload-pre-processing" containerID="cri-o://3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975" gracePeriod=30 Sep 10 17:40:13.893940 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.893910 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:40:13.898153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.898124 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:40:13.998767 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.998729 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qjf8\" (UniqueName: \"kubernetes.io/projected/82074f68-b039-4f88-bc07-45f224b28f11-kube-api-access-7qjf8\") pod \"82074f68-b039-4f88-bc07-45f224b28f11\" (UID: \"82074f68-b039-4f88-bc07-45f224b28f11\") " Sep 10 17:40:13.998966 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.998779 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2srv\" (UniqueName: \"kubernetes.io/projected/aa11b2fb-b949-4f5c-bafd-c50699aa050a-kube-api-access-s2srv\") pod \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\" (UID: \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\") " Sep 10 17:40:13.998966 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.998823 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa11b2fb-b949-4f5c-bafd-c50699aa050a-plugins-config-volume\") pod \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\" (UID: \"aa11b2fb-b949-4f5c-bafd-c50699aa050a\") " Sep 10 17:40:13.998966 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.998880 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82074f68-b039-4f88-bc07-45f224b28f11-plugins-config-volume\") pod \"82074f68-b039-4f88-bc07-45f224b28f11\" (UID: \"82074f68-b039-4f88-bc07-45f224b28f11\") " Sep 10 17:40:13.999281 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.999247 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa11b2fb-b949-4f5c-bafd-c50699aa050a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa11b2fb-b949-4f5c-bafd-c50699aa050a" (UID: "aa11b2fb-b949-4f5c-bafd-c50699aa050a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:40:13.999380 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.999310 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82074f68-b039-4f88-bc07-45f224b28f11-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82074f68-b039-4f88-bc07-45f224b28f11" (UID: "82074f68-b039-4f88-bc07-45f224b28f11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:40:13.999717 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.999694 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa11b2fb-b949-4f5c-bafd-c50699aa050a-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:13.999824 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:13.999721 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82074f68-b039-4f88-bc07-45f224b28f11-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:14.001086 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.001064 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa11b2fb-b949-4f5c-bafd-c50699aa050a-kube-api-access-s2srv" (OuterVolumeSpecName: "kube-api-access-s2srv") pod "aa11b2fb-b949-4f5c-bafd-c50699aa050a" (UID: "aa11b2fb-b949-4f5c-bafd-c50699aa050a"). InnerVolumeSpecName "kube-api-access-s2srv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:40:14.001421 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.001400 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82074f68-b039-4f88-bc07-45f224b28f11-kube-api-access-7qjf8" (OuterVolumeSpecName: "kube-api-access-7qjf8") pod "82074f68-b039-4f88-bc07-45f224b28f11" (UID: "82074f68-b039-4f88-bc07-45f224b28f11"). InnerVolumeSpecName "kube-api-access-7qjf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:40:14.100760 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.100722 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qjf8\" (UniqueName: \"kubernetes.io/projected/82074f68-b039-4f88-bc07-45f224b28f11-kube-api-access-7qjf8\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:14.100760 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.100750 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2srv\" (UniqueName: \"kubernetes.io/projected/aa11b2fb-b949-4f5c-bafd-c50699aa050a-kube-api-access-s2srv\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:14.144334 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.144248 2557 generic.go:356] "Generic (PLEG): container finished" podID="aa11b2fb-b949-4f5c-bafd-c50699aa050a" containerID="32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7" exitCode=0 Sep 10 17:40:14.144334 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.144306 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" Sep 10 17:40:14.144544 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.144337 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" event={"ID":"aa11b2fb-b949-4f5c-bafd-c50699aa050a","Type":"ContainerDied","Data":"32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7"} Sep 10 17:40:14.144544 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.144395 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94" event={"ID":"aa11b2fb-b949-4f5c-bafd-c50699aa050a","Type":"ContainerDied","Data":"f53518f4f47b8a8ed06cbc462127d9b3173de8fa840ae7115604500479a7156b"} Sep 10 17:40:14.144544 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.144424 2557 scope.go:117] "RemoveContainer" containerID="32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7" Sep 10 17:40:14.146194 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.146170 2557 generic.go:356] "Generic (PLEG): container finished" podID="82074f68-b039-4f88-bc07-45f224b28f11" containerID="3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975" exitCode=0 Sep 10 17:40:14.146321 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.146252 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" event={"ID":"82074f68-b039-4f88-bc07-45f224b28f11","Type":"ContainerDied","Data":"3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975"} Sep 10 17:40:14.146321 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.146292 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" event={"ID":"82074f68-b039-4f88-bc07-45f224b28f11","Type":"ContainerDied","Data":"4e385fb011a73b93a247e93e666576da79d48f305a5cc69edb27b81c136e1d53"} Sep 10 17:40:14.146410 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.146355 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p" Sep 10 17:40:14.155608 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.155591 2557 scope.go:117] "RemoveContainer" containerID="32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7" Sep 10 17:40:14.155900 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:40:14.155877 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7\": container with ID starting with 32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7 not found: ID does not exist" containerID="32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7" Sep 10 17:40:14.155965 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.155912 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7"} err="failed to get container status \"32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7\": rpc error: code = NotFound desc = could not find container \"32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7\": container with ID starting with 32d510cea77e2ac634b98a483ea2c35bd3b1ac175126d7785abced1c937895e7 not found: ID does not exist" Sep 10 17:40:14.155965 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.155939 2557 scope.go:117] "RemoveContainer" containerID="3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975" Sep 10 17:40:14.164222 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.164203 2557 scope.go:117] "RemoveContainer" containerID="3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975" Sep 10 17:40:14.164487 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:40:14.164464 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975\": container with ID starting with 3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975 not found: ID does not exist" containerID="3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975" Sep 10 17:40:14.164550 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.164498 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975"} err="failed to get container status \"3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975\": rpc error: code = NotFound desc = could not find container \"3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975\": container with ID starting with 3163061eee7169cec23b88d790676d09f179525be9293f53e7ed2324c7399975 not found: ID does not exist" Sep 10 17:40:14.171218 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.171197 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94"] Sep 10 17:40:14.177293 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.177269 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-52f72f-5d5dbf4b85-sgf94"] Sep 10 17:40:14.186664 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.186646 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p"] Sep 10 17:40:14.190904 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:14.190885 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-52f72f-6c446dc74b-mzl4p"] Sep 10 17:40:15.388127 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:15.388087 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82074f68-b039-4f88-bc07-45f224b28f11" path="/var/lib/kubelet/pods/82074f68-b039-4f88-bc07-45f224b28f11/volumes" Sep 10 17:40:15.388693 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:15.388671 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa11b2fb-b949-4f5c-bafd-c50699aa050a" path="/var/lib/kubelet/pods/aa11b2fb-b949-4f5c-bafd-c50699aa050a/volumes" Sep 10 17:40:20.783236 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:20.783197 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx"] Sep 10 17:40:20.783658 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:20.783425 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" containerID="cri-o://6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77" gracePeriod=30 Sep 10 17:40:26.042928 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.042904 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:40:26.207093 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.206982 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-socket\") pod \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " Sep 10 17:40:26.207093 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207029 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-data\") pod \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " Sep 10 17:40:26.207093 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207089 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istiod-ca-cert\") pod \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " Sep 10 17:40:26.207371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207123 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-envoy\") pod \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " Sep 10 17:40:26.207371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207148 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjjnk\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-kube-api-access-gjjnk\") pod \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " Sep 10 17:40:26.207371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207195 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-credential-socket\") pod \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " Sep 10 17:40:26.207371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207225 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-certs\") pod \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " Sep 10 17:40:26.207371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207250 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-podinfo\") pod \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " Sep 10 17:40:26.207371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207299 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-token\") pod \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\" (UID: \"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1\") " Sep 10 17:40:26.207371 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207314 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" (UID: "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:26.207739 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207387 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" (UID: "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:26.207739 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207561 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" (UID: "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:26.207739 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207598 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:26.207739 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207612 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:26.207936 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207772 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" (UID: "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:40:26.207936 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.207867 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" (UID: "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:26.209069 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.209026 2557 generic.go:356] "Generic (PLEG): container finished" podID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerID="6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77" exitCode=0 Sep 10 17:40:26.209204 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.209108 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" event={"ID":"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1","Type":"ContainerDied","Data":"6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77"} Sep 10 17:40:26.209204 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.209165 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" event={"ID":"f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1","Type":"ContainerDied","Data":"3766b87d7209dccf193c744e7f7ecd8f111c58514659468616e2321444f32a61"} Sep 10 17:40:26.209204 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.209193 2557 scope.go:117] "RemoveContainer" containerID="6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77" Sep 10 17:40:26.209385 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.209113 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx" Sep 10 17:40:26.209851 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.209825 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" (UID: "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:40:26.209956 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.209848 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" (UID: "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:40:26.210129 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.210104 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" (UID: "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:40:26.210129 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.210105 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-kube-api-access-gjjnk" (OuterVolumeSpecName: "kube-api-access-gjjnk") pod "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" (UID: "f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1"). InnerVolumeSpecName "kube-api-access-gjjnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:40:26.227058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.227025 2557 scope.go:117] "RemoveContainer" containerID="5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10" Sep 10 17:40:26.236631 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.236609 2557 scope.go:117] "RemoveContainer" containerID="6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77" Sep 10 17:40:26.237132 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:40:26.237098 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77\": container with ID starting with 6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77 not found: ID does not exist" containerID="6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77" Sep 10 17:40:26.237186 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.237138 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77"} err="failed to get container status \"6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77\": rpc error: code = NotFound desc = could not find container \"6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77\": container with ID starting with 6eb8057f02faef5fad916cf1bd94d6d0013974c4f39819046cb98922685feb77 not found: ID does not exist" Sep 10 17:40:26.237186 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.237162 2557 scope.go:117] "RemoveContainer" containerID="5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10" Sep 10 17:40:26.237494 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:40:26.237471 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10\": container with ID starting with 5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10 not found: ID does not exist" containerID="5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10" Sep 10 17:40:26.237581 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.237499 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10"} err="failed to get container status \"5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10\": rpc error: code = NotFound desc = could not find container \"5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10\": container with ID starting with 5f989b8962857dfd4c5fbd3041b0626d531b2a74e48794b9f53001311c85db10 not found: ID does not exist" Sep 10 17:40:26.308835 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.308809 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:26.308835 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.308831 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:26.308835 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.308839 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:26.309069 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.308847 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjnk\" (UniqueName: \"kubernetes.io/projected/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-kube-api-access-gjjnk\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:26.309069 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.308855 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:26.309069 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.308866 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:26.309069 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.308874 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:26.535520 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.535479 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx"] Sep 10 17:40:26.535691 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:26.535540 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-52f72f-openshift-default-57569775bb-xg9xx"] Sep 10 17:40:27.387473 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:27.387438 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" path="/var/lib/kubelet/pods/f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1/volumes" Sep 10 17:40:31.844060 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:31.844000 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54"] Sep 10 17:40:31.844512 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:31.844261 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" podUID="00359170-6139-4167-a6ce-d76b3afef2e3" containerName="payload-processing" containerID="cri-o://d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b" gracePeriod=30 Sep 10 17:40:31.851504 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:31.851477 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4"] Sep 10 17:40:31.856071 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:31.852315 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" podUID="b2608bac-782a-4b49-9256-464de2d889dd" containerName="payload-pre-processing" containerID="cri-o://55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c" gracePeriod=30 Sep 10 17:40:32.145430 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.145395 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:40:32.148202 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.148168 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:40:32.169945 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.169919 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zdgl\" (UniqueName: \"kubernetes.io/projected/00359170-6139-4167-a6ce-d76b3afef2e3-kube-api-access-8zdgl\") pod \"00359170-6139-4167-a6ce-d76b3afef2e3\" (UID: \"00359170-6139-4167-a6ce-d76b3afef2e3\") " Sep 10 17:40:32.170087 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.169957 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2608bac-782a-4b49-9256-464de2d889dd-plugins-config-volume\") pod \"b2608bac-782a-4b49-9256-464de2d889dd\" (UID: \"b2608bac-782a-4b49-9256-464de2d889dd\") " Sep 10 17:40:32.170087 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.170062 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf8rk\" (UniqueName: \"kubernetes.io/projected/b2608bac-782a-4b49-9256-464de2d889dd-kube-api-access-wf8rk\") pod \"b2608bac-782a-4b49-9256-464de2d889dd\" (UID: \"b2608bac-782a-4b49-9256-464de2d889dd\") " Sep 10 17:40:32.170196 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.170147 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00359170-6139-4167-a6ce-d76b3afef2e3-plugins-config-volume\") pod \"00359170-6139-4167-a6ce-d76b3afef2e3\" (UID: \"00359170-6139-4167-a6ce-d76b3afef2e3\") " Sep 10 17:40:32.170413 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.170381 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2608bac-782a-4b49-9256-464de2d889dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2608bac-782a-4b49-9256-464de2d889dd" (UID: "b2608bac-782a-4b49-9256-464de2d889dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:40:32.170518 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.170496 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2608bac-782a-4b49-9256-464de2d889dd-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:32.170587 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.170516 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00359170-6139-4167-a6ce-d76b3afef2e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00359170-6139-4167-a6ce-d76b3afef2e3" (UID: "00359170-6139-4167-a6ce-d76b3afef2e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:40:32.172139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.172111 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2608bac-782a-4b49-9256-464de2d889dd-kube-api-access-wf8rk" (OuterVolumeSpecName: "kube-api-access-wf8rk") pod "b2608bac-782a-4b49-9256-464de2d889dd" (UID: "b2608bac-782a-4b49-9256-464de2d889dd"). InnerVolumeSpecName "kube-api-access-wf8rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:40:32.172224 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.172189 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00359170-6139-4167-a6ce-d76b3afef2e3-kube-api-access-8zdgl" (OuterVolumeSpecName: "kube-api-access-8zdgl") pod "00359170-6139-4167-a6ce-d76b3afef2e3" (UID: "00359170-6139-4167-a6ce-d76b3afef2e3"). InnerVolumeSpecName "kube-api-access-8zdgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:40:32.240888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.240851 2557 generic.go:356] "Generic (PLEG): container finished" podID="00359170-6139-4167-a6ce-d76b3afef2e3" containerID="d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b" exitCode=0 Sep 10 17:40:32.241081 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.240884 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" event={"ID":"00359170-6139-4167-a6ce-d76b3afef2e3","Type":"ContainerDied","Data":"d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b"} Sep 10 17:40:32.241081 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.240901 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" Sep 10 17:40:32.241081 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.240979 2557 scope.go:117] "RemoveContainer" containerID="d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b" Sep 10 17:40:32.241081 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.240966 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54" event={"ID":"00359170-6139-4167-a6ce-d76b3afef2e3","Type":"ContainerDied","Data":"4be04119a05bc0f4dd7886a5cb6b40df5a172800c2cf0f9d7d27334a072ee3b7"} Sep 10 17:40:32.242288 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.242266 2557 generic.go:356] "Generic (PLEG): container finished" podID="b2608bac-782a-4b49-9256-464de2d889dd" containerID="55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c" exitCode=0 Sep 10 17:40:32.242396 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.242318 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" Sep 10 17:40:32.242396 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.242337 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" event={"ID":"b2608bac-782a-4b49-9256-464de2d889dd","Type":"ContainerDied","Data":"55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c"} Sep 10 17:40:32.242396 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.242372 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4" event={"ID":"b2608bac-782a-4b49-9256-464de2d889dd","Type":"ContainerDied","Data":"c32a6a84e82dd6f2825f54c763df3bd2083cac1dca53a060642bacbc0f02fd40"} Sep 10 17:40:32.251803 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.251695 2557 scope.go:117] "RemoveContainer" containerID="d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b" Sep 10 17:40:32.252019 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:40:32.251992 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b\": container with ID starting with d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b not found: ID does not exist" containerID="d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b" Sep 10 17:40:32.252120 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.252028 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b"} err="failed to get container status \"d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b\": rpc error: code = NotFound desc = could not find container \"d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b\": container with ID starting with d08c79533cc1632ebe7ab2c01c9977f453a3f2e4003d6278f97fd875f481264b not found: ID does not exist" Sep 10 17:40:32.252120 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.252082 2557 scope.go:117] "RemoveContainer" containerID="55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c" Sep 10 17:40:32.260192 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.260174 2557 scope.go:117] "RemoveContainer" containerID="55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c" Sep 10 17:40:32.260453 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:40:32.260435 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c\": container with ID starting with 55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c not found: ID does not exist" containerID="55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c" Sep 10 17:40:32.260506 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.260460 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c"} err="failed to get container status \"55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c\": rpc error: code = NotFound desc = could not find container \"55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c\": container with ID starting with 55770395671d38ac393f8c3e62e1ef44c3b2e6c2a90ed8bf6193d1af5b8d135c not found: ID does not exist" Sep 10 17:40:32.266416 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.266396 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54"] Sep 10 17:40:32.271488 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.271398 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8rk\" (UniqueName: \"kubernetes.io/projected/b2608bac-782a-4b49-9256-464de2d889dd-kube-api-access-wf8rk\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:32.271488 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.271438 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00359170-6139-4167-a6ce-d76b3afef2e3-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:32.271488 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.271455 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdgl\" (UniqueName: \"kubernetes.io/projected/00359170-6139-4167-a6ce-d76b3afef2e3-kube-api-access-8zdgl\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:40:32.272977 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.272956 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5912a3-9dc878549-kvf54"] Sep 10 17:40:32.282437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.282414 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4"] Sep 10 17:40:32.285258 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:32.285234 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5912a3-bf47b5d9d-p5qs4"] Sep 10 17:40:33.386058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:33.386010 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00359170-6139-4167-a6ce-d76b3afef2e3" path="/var/lib/kubelet/pods/00359170-6139-4167-a6ce-d76b3afef2e3/volumes" Sep 10 17:40:33.386582 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:33.386559 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2608bac-782a-4b49-9256-464de2d889dd" path="/var/lib/kubelet/pods/b2608bac-782a-4b49-9256-464de2d889dd/volumes" Sep 10 17:40:35.914268 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914232 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7"] Sep 10 17:40:35.914762 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914742 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11602a01-c1c5-4ce7-806c-7bc1ba1625c3" containerName="payload-pre-processing" Sep 10 17:40:35.914837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914765 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="11602a01-c1c5-4ce7-806c-7bc1ba1625c3" containerName="payload-pre-processing" Sep 10 17:40:35.914837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914781 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2608bac-782a-4b49-9256-464de2d889dd" containerName="payload-pre-processing" Sep 10 17:40:35.914837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914790 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2608bac-782a-4b49-9256-464de2d889dd" containerName="payload-pre-processing" Sep 10 17:40:35.914837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914801 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82074f68-b039-4f88-bc07-45f224b28f11" containerName="payload-pre-processing" Sep 10 17:40:35.914837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914810 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="82074f68-b039-4f88-bc07-45f224b28f11" containerName="payload-pre-processing" Sep 10 17:40:35.914837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914822 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00359170-6139-4167-a6ce-d76b3afef2e3" containerName="payload-processing" Sep 10 17:40:35.914837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914830 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="00359170-6139-4167-a6ce-d76b3afef2e3" containerName="payload-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914843 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914852 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914868 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f" containerName="payload-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914876 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f" containerName="payload-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914885 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914895 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914909 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa11b2fb-b949-4f5c-bafd-c50699aa050a" containerName="payload-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914917 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa11b2fb-b949-4f5c-bafd-c50699aa050a" containerName="payload-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914930 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914938 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914947 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914955 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914965 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914974 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914984 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.914992 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915104 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915116 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f313f8aa-348e-41eb-9c0a-0f2dc18ab1b1" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915125 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2608bac-782a-4b49-9256-464de2d889dd" containerName="payload-pre-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915136 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="11602a01-c1c5-4ce7-806c-7bc1ba1625c3" containerName="payload-pre-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915146 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915154 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="00359170-6139-4167-a6ce-d76b3afef2e3" containerName="payload-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915162 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa11b2fb-b949-4f5c-bafd-c50699aa050a" containerName="payload-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915171 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915181 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="82074f68-b039-4f88-bc07-45f224b28f11" containerName="payload-pre-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915189 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f51e6b4-d07c-4c2a-82fb-d15f1cf3640f" containerName="payload-processing" Sep 10 17:40:35.915221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.915198 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a4fc89-6e21-4c75-9916-b70da82202d9" containerName="istio-proxy" Sep 10 17:40:35.919887 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.919866 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:35.922065 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.922046 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c1fb61\"/\"openshift-service-ca.crt\"" Sep 10 17:40:35.922766 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.922746 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c1fb61\"/\"default-dockercfg-jc8b9\"" Sep 10 17:40:35.922831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.922762 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c1fb61\"/\"rate-test-model-c1fb61-kserve-self-signed-certs\"" Sep 10 17:40:35.922831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.922776 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c1fb61\"/\"kube-root-ca.crt\"" Sep 10 17:40:35.927090 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:35.927071 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7"] Sep 10 17:40:36.002370 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.002343 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8g9d\" (UniqueName: \"kubernetes.io/projected/dad09839-0b81-43ba-90e4-9e8cce6c92a5-kube-api-access-z8g9d\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.002495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.002376 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tmp-dir\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.002495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.002394 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-dshm\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.002495 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.002413 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tls-certs\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.002637 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.002530 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-home\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.002637 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.002550 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-model-cache\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.103610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.103581 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-home\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.103610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.103610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-model-cache\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.103794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.103642 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8g9d\" (UniqueName: \"kubernetes.io/projected/dad09839-0b81-43ba-90e4-9e8cce6c92a5-kube-api-access-z8g9d\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.103794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.103666 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tmp-dir\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.103794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.103689 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-dshm\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.103794 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.103715 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tls-certs\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.104077 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.104028 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-model-cache\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.104077 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.104067 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tmp-dir\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.104196 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.104118 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-home\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.106000 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.105980 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-dshm\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.106277 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.106259 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tls-certs\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.111210 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.111189 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8g9d\" (UniqueName: \"kubernetes.io/projected/dad09839-0b81-43ba-90e4-9e8cce6c92a5-kube-api-access-z8g9d\") pod \"rate-test-model-c1fb61-kserve-f96657ff-pdkf7\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.229505 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.229429 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:36.503354 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:36.503298 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7"] Sep 10 17:40:36.505413 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:40:36.505387 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddad09839_0b81_43ba_90e4_9e8cce6c92a5.slice/crio-f4f2889e4086a9568dea43b18cb955d8ab8d925fba4e0b1b1ccebf32eb499316 WatchSource:0}: Error finding container f4f2889e4086a9568dea43b18cb955d8ab8d925fba4e0b1b1ccebf32eb499316: Status 404 returned error can't find the container with id f4f2889e4086a9568dea43b18cb955d8ab8d925fba4e0b1b1ccebf32eb499316 Sep 10 17:40:37.271554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:37.271519 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" event={"ID":"dad09839-0b81-43ba-90e4-9e8cce6c92a5","Type":"ContainerStarted","Data":"8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522"} Sep 10 17:40:37.271554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:37.271558 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" event={"ID":"dad09839-0b81-43ba-90e4-9e8cce6c92a5","Type":"ContainerStarted","Data":"f4f2889e4086a9568dea43b18cb955d8ab8d925fba4e0b1b1ccebf32eb499316"} Sep 10 17:40:37.292060 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:37.291999 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" podStartSLOduration=2.291984541 podStartE2EDuration="2.291984541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:40:37.289973484 +0000 UTC m=+1636.387871844" watchObservedRunningTime="2026-09-10 17:40:37.291984541 +0000 UTC m=+1636.389882901" Sep 10 17:40:46.230360 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:46.230317 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:46.230925 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:46.230399 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:46.242737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:46.242717 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:40:46.327001 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:40:46.326979 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:41:18.436110 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:18.436080 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7"] Sep 10 17:41:19.162738 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.162700 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg"] Sep 10 17:41:19.163342 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.163318 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e11f413-5d01-4ec0-8d69-f39d4da9e6ec" containerName="istio-proxy" Sep 10 17:41:19.166486 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.166468 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.169301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.169278 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7858f0-openshift-default-dockercfg-cj9wv\"" Sep 10 17:41:19.184153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.184128 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg"] Sep 10 17:41:19.343538 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.343503 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-podinfo\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.343724 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.343546 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjjlf\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-kube-api-access-fjjlf\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.343724 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.343569 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-envoy\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.343724 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.343638 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-token\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.343724 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.343683 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-credential-socket\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.343724 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.343702 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-data\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.343918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.343741 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-socket\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.343918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.343794 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-certs\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.343918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.343835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istiod-ca-cert\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.444290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444220 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-podinfo\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.444290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444258 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjjlf\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-kube-api-access-fjjlf\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.444749 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-envoy\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.444749 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444328 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-token\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.444749 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444386 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-credential-socket\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.444749 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444413 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-data\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.444749 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-socket\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.444749 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444496 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-certs\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.444749 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444545 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istiod-ca-cert\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.445132 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444763 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-credential-socket\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.445132 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444911 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-data\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.445132 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444951 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-socket\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.445132 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.444991 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-certs\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.445404 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.445382 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istiod-ca-cert\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.446900 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.446876 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-podinfo\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.446992 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.446935 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-envoy\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.451953 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.451926 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjjlf\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-kube-api-access-fjjlf\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.452372 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.452352 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-token\") pod \"e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.477265 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.477241 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:19.758146 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.758122 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg"] Sep 10 17:41:19.759883 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:41:19.759853 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a49cb4_ec56_4581_9b0f_93c60d2ac6bf.slice/crio-038bcf2c4b1512d2bdea4d93846ae2beae6f2905515441df98a1f701ee545f1d WatchSource:0}: Error finding container 038bcf2c4b1512d2bdea4d93846ae2beae6f2905515441df98a1f701ee545f1d: Status 404 returned error can't find the container with id 038bcf2c4b1512d2bdea4d93846ae2beae6f2905515441df98a1f701ee545f1d Sep 10 17:41:19.762019 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.761983 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:41:19.762116 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.762068 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:41:19.762116 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:19.762099 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:41:20.477258 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:20.477221 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" event={"ID":"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf","Type":"ContainerStarted","Data":"35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d"} Sep 10 17:41:20.477627 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:20.477281 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" event={"ID":"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf","Type":"ContainerStarted","Data":"038bcf2c4b1512d2bdea4d93846ae2beae6f2905515441df98a1f701ee545f1d"} Sep 10 17:41:20.477627 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:20.477384 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:20.482276 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:20.482256 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:20.503720 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:20.503681 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" podStartSLOduration=1.5036700600000001 podStartE2EDuration="1.50367006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:20.501916873 +0000 UTC m=+1679.599815248" watchObservedRunningTime="2026-09-10 17:41:20.50367006 +0000 UTC m=+1679.601568420" Sep 10 17:41:21.482097 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:21.482069 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:21.483107 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:21.483086 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:25.158697 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.158664 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g"] Sep 10 17:41:25.161988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.161972 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:25.164192 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.164174 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7858f0\"" Sep 10 17:41:25.175651 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.175626 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g"] Sep 10 17:41:25.186922 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.186899 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8"] Sep 10 17:41:25.190862 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.190840 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:25.192732 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.192711 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7858f0-dockercfg-jzbm7\"" Sep 10 17:41:25.200769 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.200745 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8"] Sep 10 17:41:25.307516 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.307464 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8\" (UID: \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:25.307730 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.307525 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjwb\" (UniqueName: \"kubernetes.io/projected/0835426f-455a-4fa3-a63a-844a332bb0fc-kube-api-access-pwjwb\") pod \"payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g\" (UID: \"0835426f-455a-4fa3-a63a-844a332bb0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:25.307730 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.307631 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835426f-455a-4fa3-a63a-844a332bb0fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g\" (UID: \"0835426f-455a-4fa3-a63a-844a332bb0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:25.307730 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.307668 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ccr\" (UniqueName: \"kubernetes.io/projected/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-kube-api-access-v2ccr\") pod \"payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8\" (UID: \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:25.408344 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.408316 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8\" (UID: \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:25.408497 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.408359 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjwb\" (UniqueName: \"kubernetes.io/projected/0835426f-455a-4fa3-a63a-844a332bb0fc-kube-api-access-pwjwb\") pod \"payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g\" (UID: \"0835426f-455a-4fa3-a63a-844a332bb0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:25.408497 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.408454 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835426f-455a-4fa3-a63a-844a332bb0fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g\" (UID: \"0835426f-455a-4fa3-a63a-844a332bb0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:25.408632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.408503 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ccr\" (UniqueName: \"kubernetes.io/projected/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-kube-api-access-v2ccr\") pod \"payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8\" (UID: \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:25.409194 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.409135 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8\" (UID: \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:25.409305 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.409248 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835426f-455a-4fa3-a63a-844a332bb0fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g\" (UID: \"0835426f-455a-4fa3-a63a-844a332bb0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:25.420147 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.420127 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ccr\" (UniqueName: \"kubernetes.io/projected/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-kube-api-access-v2ccr\") pod \"payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8\" (UID: \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:25.420244 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.420148 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjwb\" (UniqueName: \"kubernetes.io/projected/0835426f-455a-4fa3-a63a-844a332bb0fc-kube-api-access-pwjwb\") pod \"payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g\" (UID: \"0835426f-455a-4fa3-a63a-844a332bb0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:25.471594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.471569 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:25.502451 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.502200 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:25.767535 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.767510 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g"] Sep 10 17:41:25.769128 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:41:25.769094 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0835426f_455a_4fa3_a63a_844a332bb0fc.slice/crio-7c630802ed802627a2b22eb277659ebf47ea84663bd07c0d380e2db6ce0de1ab WatchSource:0}: Error finding container 7c630802ed802627a2b22eb277659ebf47ea84663bd07c0d380e2db6ce0de1ab: Status 404 returned error can't find the container with id 7c630802ed802627a2b22eb277659ebf47ea84663bd07c0d380e2db6ce0de1ab Sep 10 17:41:25.791280 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:25.791261 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8"] Sep 10 17:41:25.801016 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:41:25.800988 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf629b6_dcd8_4875_b1c6_cb4c8de076c4.slice/crio-59e4fb8e91473a39982eccaae24ed054702173b14c554d30f10e14d8d105704d WatchSource:0}: Error finding container 59e4fb8e91473a39982eccaae24ed054702173b14c554d30f10e14d8d105704d: Status 404 returned error can't find the container with id 59e4fb8e91473a39982eccaae24ed054702173b14c554d30f10e14d8d105704d Sep 10 17:41:26.520779 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:26.520737 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" event={"ID":"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4","Type":"ContainerStarted","Data":"b4259d24e8ef8dcf8ef78ef8726598c8fe30ca546d598d10277d8f5ec8f655e2"} Sep 10 17:41:26.521237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:26.521156 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" event={"ID":"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4","Type":"ContainerStarted","Data":"59e4fb8e91473a39982eccaae24ed054702173b14c554d30f10e14d8d105704d"} Sep 10 17:41:26.521237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:26.521198 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:26.522644 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:26.522617 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" event={"ID":"0835426f-455a-4fa3-a63a-844a332bb0fc","Type":"ContainerStarted","Data":"ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5"} Sep 10 17:41:26.522746 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:26.522669 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" event={"ID":"0835426f-455a-4fa3-a63a-844a332bb0fc","Type":"ContainerStarted","Data":"7c630802ed802627a2b22eb277659ebf47ea84663bd07c0d380e2db6ce0de1ab"} Sep 10 17:41:26.522791 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:26.522744 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:26.549124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:26.549082 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" podStartSLOduration=1.5490677160000001 podStartE2EDuration="1.549067716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:26.548271266 +0000 UTC m=+1685.646169628" watchObservedRunningTime="2026-09-10 17:41:26.549067716 +0000 UTC m=+1685.646966076" Sep 10 17:41:26.566499 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:26.566452 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" podStartSLOduration=1.5664352049999999 podStartE2EDuration="1.566435205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:26.563828547 +0000 UTC m=+1685.661726907" watchObservedRunningTime="2026-09-10 17:41:26.566435205 +0000 UTC m=+1685.664333571" Sep 10 17:41:33.465288 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.465222 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" podUID="dad09839-0b81-43ba-90e4-9e8cce6c92a5" containerName="main" containerID="cri-o://8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522" gracePeriod=45 Sep 10 17:41:33.726106 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.726055 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:41:33.784429 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.784404 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-dshm\") pod \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " Sep 10 17:41:33.784618 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.784442 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tls-certs\") pod \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " Sep 10 17:41:33.784618 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.784495 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8g9d\" (UniqueName: \"kubernetes.io/projected/dad09839-0b81-43ba-90e4-9e8cce6c92a5-kube-api-access-z8g9d\") pod \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " Sep 10 17:41:33.784618 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.784528 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tmp-dir\") pod \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " Sep 10 17:41:33.784618 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.784559 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-home\") pod \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " Sep 10 17:41:33.784618 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.784608 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-model-cache\") pod \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\" (UID: \"dad09839-0b81-43ba-90e4-9e8cce6c92a5\") " Sep 10 17:41:33.784941 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.784909 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dad09839-0b81-43ba-90e4-9e8cce6c92a5" (UID: "dad09839-0b81-43ba-90e4-9e8cce6c92a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:41:33.785062 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.785024 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-home" (OuterVolumeSpecName: "home") pod "dad09839-0b81-43ba-90e4-9e8cce6c92a5" (UID: "dad09839-0b81-43ba-90e4-9e8cce6c92a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:41:33.785142 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.785074 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "dad09839-0b81-43ba-90e4-9e8cce6c92a5" (UID: "dad09839-0b81-43ba-90e4-9e8cce6c92a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:41:33.786737 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.786711 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dad09839-0b81-43ba-90e4-9e8cce6c92a5" (UID: "dad09839-0b81-43ba-90e4-9e8cce6c92a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:41:33.786839 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.786765 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad09839-0b81-43ba-90e4-9e8cce6c92a5-kube-api-access-z8g9d" (OuterVolumeSpecName: "kube-api-access-z8g9d") pod "dad09839-0b81-43ba-90e4-9e8cce6c92a5" (UID: "dad09839-0b81-43ba-90e4-9e8cce6c92a5"). InnerVolumeSpecName "kube-api-access-z8g9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:41:33.786934 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.786918 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-dshm" (OuterVolumeSpecName: "dshm") pod "dad09839-0b81-43ba-90e4-9e8cce6c92a5" (UID: "dad09839-0b81-43ba-90e4-9e8cce6c92a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:41:33.885600 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.885574 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-dshm\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:41:33.885600 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.885599 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tls-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:41:33.885740 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.885610 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8g9d\" (UniqueName: \"kubernetes.io/projected/dad09839-0b81-43ba-90e4-9e8cce6c92a5-kube-api-access-z8g9d\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:41:33.885740 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.885650 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-tmp-dir\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:41:33.885740 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.885659 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-home\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:41:33.885740 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:33.885666 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dad09839-0b81-43ba-90e4-9e8cce6c92a5-model-cache\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:41:34.568642 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:34.568601 2557 generic.go:356] "Generic (PLEG): container finished" podID="dad09839-0b81-43ba-90e4-9e8cce6c92a5" containerID="8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522" exitCode=0 Sep 10 17:41:34.569131 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:34.568646 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" Sep 10 17:41:34.569131 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:34.568689 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" event={"ID":"dad09839-0b81-43ba-90e4-9e8cce6c92a5","Type":"ContainerDied","Data":"8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522"} Sep 10 17:41:34.569131 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:34.568740 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7" event={"ID":"dad09839-0b81-43ba-90e4-9e8cce6c92a5","Type":"ContainerDied","Data":"f4f2889e4086a9568dea43b18cb955d8ab8d925fba4e0b1b1ccebf32eb499316"} Sep 10 17:41:34.569131 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:34.568764 2557 scope.go:117] "RemoveContainer" containerID="8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522" Sep 10 17:41:34.578413 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:34.578394 2557 scope.go:117] "RemoveContainer" containerID="8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522" Sep 10 17:41:34.578668 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:41:34.578650 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522\": container with ID starting with 8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522 not found: ID does not exist" containerID="8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522" Sep 10 17:41:34.578724 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:34.578681 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522"} err="failed to get container status \"8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522\": rpc error: code = NotFound desc = could not find container \"8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522\": container with ID starting with 8fec1ed9414068ca5da63012b38a95c8e4f9dfc900de4ae7e0472eec48eee522 not found: ID does not exist" Sep 10 17:41:34.590593 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:34.590573 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7"] Sep 10 17:41:34.597074 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:34.597052 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c1fb61/rate-test-model-c1fb61-kserve-f96657ff-pdkf7"] Sep 10 17:41:35.386317 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:35.386249 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad09839-0b81-43ba-90e4-9e8cce6c92a5" path="/var/lib/kubelet/pods/dad09839-0b81-43ba-90e4-9e8cce6c92a5/volumes" Sep 10 17:41:37.524168 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:37.524131 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 17:41:37.524494 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:37.524188 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 17:41:37.529614 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:37.529588 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:41:37.529738 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:37.529673 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:41:38.592741 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:38.592712 2557 generic.go:356] "Generic (PLEG): container finished" podID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerID="35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d" exitCode=0 Sep 10 17:41:38.593230 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:38.592759 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" event={"ID":"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf","Type":"ContainerDied","Data":"35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d"} Sep 10 17:41:38.593230 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:38.593098 2557 scope.go:117] "RemoveContainer" containerID="35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d" Sep 10 17:41:38.594232 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:38.594189 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:41:38.594329 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:38.594273 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:41:38.594329 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:38.594316 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:41:39.599069 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:39.599018 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" event={"ID":"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf","Type":"ContainerStarted","Data":"c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05"} Sep 10 17:41:40.477975 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:40.477939 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:40.478265 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:40.478242 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 10 17:41:40.478354 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:40.478296 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 10 17:41:41.478102 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:41.478066 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 10 17:41:41.478551 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:41.478127 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 10 17:41:42.478427 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:42.478390 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 10 17:41:42.478802 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:42.478440 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 10 17:41:43.478507 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:43.478472 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 10 17:41:43.478884 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:43.478522 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 10 17:41:44.482487 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:44.482453 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:44.482949 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:44.482852 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:44.483691 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:44.483671 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:41:48.528085 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.528017 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd"] Sep 10 17:41:48.528658 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.528638 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad09839-0b81-43ba-90e4-9e8cce6c92a5" containerName="main" Sep 10 17:41:48.528744 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.528661 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad09839-0b81-43ba-90e4-9e8cce6c92a5" containerName="main" Sep 10 17:41:48.528798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.528772 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad09839-0b81-43ba-90e4-9e8cce6c92a5" containerName="main" Sep 10 17:41:48.531883 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.531863 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.535201 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.535177 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fdb1bd-openshift-default-dockercfg-sjhhn\"" Sep 10 17:41:48.551561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.551535 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd"] Sep 10 17:41:48.613446 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.613414 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48cxn\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-kube-api-access-48cxn\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.613580 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.613472 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-envoy\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.613580 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.613536 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-certs\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.613580 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.613567 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-token\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.613755 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.613588 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-podinfo\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.613755 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.613617 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f9e963-90ce-43c3-a35c-ce9e164117ae-istiod-ca-cert\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.613755 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.613729 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-data\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.613910 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.613779 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-socket\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.613910 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.613806 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-credential-socket\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.714814 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.714788 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-data\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.714968 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.714819 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-socket\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.714968 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.714838 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-credential-socket\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.714968 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.714894 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48cxn\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-kube-api-access-48cxn\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.714968 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.714919 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-envoy\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.714968 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.714952 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-certs\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.715224 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.714979 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-token\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.715224 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.715004 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-podinfo\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.715224 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.715050 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f9e963-90ce-43c3-a35c-ce9e164117ae-istiod-ca-cert\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.715224 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.715214 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-data\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.715424 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.715242 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-socket\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.715424 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.715348 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-credential-socket\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.715424 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.715392 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-certs\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.715675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.715652 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f9e963-90ce-43c3-a35c-ce9e164117ae-istiod-ca-cert\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.716965 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.716946 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-envoy\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.717438 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.717417 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-podinfo\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.733301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.733273 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-token\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.733401 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.733340 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48cxn\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-kube-api-access-48cxn\") pod \"e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:48.842921 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:48.842836 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:49.134346 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:49.134305 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd"] Sep 10 17:41:49.136776 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:41:49.136748 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f9e963_90ce_43c3_a35c_ce9e164117ae.slice/crio-1fe08005dae1d7d0ec5c5d71a45e10b80aa142cd7980f5257dfc8da65ad1e40c WatchSource:0}: Error finding container 1fe08005dae1d7d0ec5c5d71a45e10b80aa142cd7980f5257dfc8da65ad1e40c: Status 404 returned error can't find the container with id 1fe08005dae1d7d0ec5c5d71a45e10b80aa142cd7980f5257dfc8da65ad1e40c Sep 10 17:41:49.138965 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:49.138936 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:41:49.139046 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:49.138996 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:41:49.139093 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:49.139055 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:41:49.654430 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:49.654396 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" event={"ID":"64f9e963-90ce-43c3-a35c-ce9e164117ae","Type":"ContainerStarted","Data":"3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208"} Sep 10 17:41:49.654939 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:49.654448 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" event={"ID":"64f9e963-90ce-43c3-a35c-ce9e164117ae","Type":"ContainerStarted","Data":"1fe08005dae1d7d0ec5c5d71a45e10b80aa142cd7980f5257dfc8da65ad1e40c"} Sep 10 17:41:49.679437 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:49.679384 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" podStartSLOduration=1.679367895 podStartE2EDuration="1.679367895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:49.676838689 +0000 UTC m=+1708.774737072" watchObservedRunningTime="2026-09-10 17:41:49.679367895 +0000 UTC m=+1708.777266254" Sep 10 17:41:49.843279 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:49.843245 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:50.849530 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:50.849504 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:51.664482 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:51.664442 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:51.665354 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:51.665334 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:41:55.497417 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.497382 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc"] Sep 10 17:41:55.500780 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.500764 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:41:55.503300 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.503282 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fdb1bd\"" Sep 10 17:41:55.503390 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.503307 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fdb1bd-dockercfg-nn476\"" Sep 10 17:41:55.511729 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.511711 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc"] Sep 10 17:41:55.564455 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.564425 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw"] Sep 10 17:41:55.568146 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.568123 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:41:55.575413 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.575376 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt2mp\" (UniqueName: \"kubernetes.io/projected/12bbd43f-40bb-464e-bf38-44590b92069a-kube-api-access-mt2mp\") pod \"payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc\" (UID: \"12bbd43f-40bb-464e-bf38-44590b92069a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:41:55.575526 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.575424 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bbd43f-40bb-464e-bf38-44590b92069a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc\" (UID: \"12bbd43f-40bb-464e-bf38-44590b92069a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:41:55.583146 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.583126 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw"] Sep 10 17:41:55.676859 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.676828 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt2mp\" (UniqueName: \"kubernetes.io/projected/12bbd43f-40bb-464e-bf38-44590b92069a-kube-api-access-mt2mp\") pod \"payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc\" (UID: \"12bbd43f-40bb-464e-bf38-44590b92069a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:41:55.677011 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.676866 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bbd43f-40bb-464e-bf38-44590b92069a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc\" (UID: \"12bbd43f-40bb-464e-bf38-44590b92069a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:41:55.677011 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.676916 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690813f-5d44-4e99-8783-7764efb4bbfe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw\" (UID: \"0690813f-5d44-4e99-8783-7764efb4bbfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:41:55.677011 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.676985 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55pb\" (UniqueName: \"kubernetes.io/projected/0690813f-5d44-4e99-8783-7764efb4bbfe-kube-api-access-n55pb\") pod \"payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw\" (UID: \"0690813f-5d44-4e99-8783-7764efb4bbfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:41:55.677610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.677588 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bbd43f-40bb-464e-bf38-44590b92069a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc\" (UID: \"12bbd43f-40bb-464e-bf38-44590b92069a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:41:55.692322 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.692293 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt2mp\" (UniqueName: \"kubernetes.io/projected/12bbd43f-40bb-464e-bf38-44590b92069a-kube-api-access-mt2mp\") pod \"payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc\" (UID: \"12bbd43f-40bb-464e-bf38-44590b92069a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:41:55.778472 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.778420 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690813f-5d44-4e99-8783-7764efb4bbfe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw\" (UID: \"0690813f-5d44-4e99-8783-7764efb4bbfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:41:55.778472 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.778464 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55pb\" (UniqueName: \"kubernetes.io/projected/0690813f-5d44-4e99-8783-7764efb4bbfe-kube-api-access-n55pb\") pod \"payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw\" (UID: \"0690813f-5d44-4e99-8783-7764efb4bbfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:41:55.779145 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.779126 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690813f-5d44-4e99-8783-7764efb4bbfe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw\" (UID: \"0690813f-5d44-4e99-8783-7764efb4bbfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:41:55.787120 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.787100 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55pb\" (UniqueName: \"kubernetes.io/projected/0690813f-5d44-4e99-8783-7764efb4bbfe-kube-api-access-n55pb\") pod \"payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw\" (UID: \"0690813f-5d44-4e99-8783-7764efb4bbfe\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:41:55.811071 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.811027 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:41:55.878618 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:55.878590 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:41:56.085839 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.085809 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc"] Sep 10 17:41:56.087218 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:41:56.087189 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bbd43f_40bb_464e_bf38_44590b92069a.slice/crio-bafe42ab6186e2845eb4528fb657734c4f030829e6f2eda6005595f1fb03a979 WatchSource:0}: Error finding container bafe42ab6186e2845eb4528fb657734c4f030829e6f2eda6005595f1fb03a979: Status 404 returned error can't find the container with id bafe42ab6186e2845eb4528fb657734c4f030829e6f2eda6005595f1fb03a979 Sep 10 17:41:56.159809 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.157302 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw"] Sep 10 17:41:56.164847 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:41:56.164824 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0690813f_5d44_4e99_8783_7764efb4bbfe.slice/crio-0712d55652aba894d658441350d4e55dc615b187cf6079779f3f751b5b1cb231 WatchSource:0}: Error finding container 0712d55652aba894d658441350d4e55dc615b187cf6079779f3f751b5b1cb231: Status 404 returned error can't find the container with id 0712d55652aba894d658441350d4e55dc615b187cf6079779f3f751b5b1cb231 Sep 10 17:41:56.691859 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.691813 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" event={"ID":"12bbd43f-40bb-464e-bf38-44590b92069a","Type":"ContainerStarted","Data":"a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a"} Sep 10 17:41:56.692328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.691995 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:41:56.692328 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.692043 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" event={"ID":"12bbd43f-40bb-464e-bf38-44590b92069a","Type":"ContainerStarted","Data":"bafe42ab6186e2845eb4528fb657734c4f030829e6f2eda6005595f1fb03a979"} Sep 10 17:41:56.693643 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.693618 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" event={"ID":"0690813f-5d44-4e99-8783-7764efb4bbfe","Type":"ContainerStarted","Data":"4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed"} Sep 10 17:41:56.693782 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.693657 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" event={"ID":"0690813f-5d44-4e99-8783-7764efb4bbfe","Type":"ContainerStarted","Data":"0712d55652aba894d658441350d4e55dc615b187cf6079779f3f751b5b1cb231"} Sep 10 17:41:56.693782 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.693771 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:41:56.717485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.717444 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" podStartSLOduration=1.717431105 podStartE2EDuration="1.717431105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:56.716009051 +0000 UTC m=+1715.813907412" watchObservedRunningTime="2026-09-10 17:41:56.717431105 +0000 UTC m=+1715.815329464" Sep 10 17:41:56.740817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:41:56.740777 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" podStartSLOduration=1.740763933 podStartE2EDuration="1.740763933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:56.738872373 +0000 UTC m=+1715.836770733" watchObservedRunningTime="2026-09-10 17:41:56.740763933 +0000 UTC m=+1715.838662293" Sep 10 17:42:07.700744 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:07.700713 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:42:07.701227 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:07.700782 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:42:07.706150 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:07.706122 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 17:42:07.706267 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:07.706167 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 17:42:09.760542 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:09.760510 2557 generic.go:356] "Generic (PLEG): container finished" podID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerID="3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208" exitCode=0 Sep 10 17:42:09.760913 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:09.760561 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" event={"ID":"64f9e963-90ce-43c3-a35c-ce9e164117ae","Type":"ContainerDied","Data":"3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208"} Sep 10 17:42:09.760913 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:09.760864 2557 scope.go:117] "RemoveContainer" containerID="3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208" Sep 10 17:42:09.761802 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:09.761769 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:42:09.761899 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:09.761855 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:42:09.761960 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:09.761905 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:42:10.771023 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:10.770986 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" event={"ID":"64f9e963-90ce-43c3-a35c-ce9e164117ae","Type":"ContainerStarted","Data":"14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1"} Sep 10 17:42:10.843189 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:10.843160 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:42:10.843485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:10.843461 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 17:42:10.843601 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:10.843508 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 17:42:11.862058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:11.858500 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 17:42:11.862058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:11.858558 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 17:42:12.843554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:12.843515 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 17:42:12.843758 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:12.843571 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 17:42:13.843588 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:13.843548 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 17:42:13.844068 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:13.843615 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 17:42:14.843508 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:14.843474 2557 patch_prober.go:28] interesting pod/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 10 17:42:14.843786 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:14.843758 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 10 17:42:15.846809 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:15.846780 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:42:15.847167 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:15.847141 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:42:15.847862 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:15.847842 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:42:21.761630 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:21.761598 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w7ck4_c21f986f-64b2-4347-af53-2d38959e718f/payload-processing/0.log" Sep 10 17:42:21.892174 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:21.892108 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw_0690813f-5d44-4e99-8783-7764efb4bbfe/payload-processing/0.log" Sep 10 17:42:57.766668 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:57.766634 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8_0cf629b6-dcd8-4875-b1c6-cb4c8de076c4/payload-processing/0.log" Sep 10 17:42:57.890990 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:57.890964 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w7ck4_c21f986f-64b2-4347-af53-2d38959e718f/payload-processing/0.log" Sep 10 17:42:58.016151 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.016124 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw_0690813f-5d44-4e99-8783-7764efb4bbfe/payload-processing/0.log" Sep 10 17:42:58.644960 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.644929 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t"] Sep 10 17:42:58.648919 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.648901 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.651187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.651170 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-8a34df-openshift-default-dockercfg-m8zhl\"" Sep 10 17:42:58.670239 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.670212 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t"] Sep 10 17:42:58.806458 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.806425 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-credential-socket\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.806888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.806470 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-certs\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.806888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.806495 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-data\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.806888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.806517 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtr8\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-kube-api-access-rxtr8\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.806888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.806579 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b201b8-05b4-4768-a9b0-bc762d71d79b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.806888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.806638 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-token\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.806888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.806673 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-envoy\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.806888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.806707 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-socket\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.806888 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.806735 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-podinfo\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907349 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907293 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-certs\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907349 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907320 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-data\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907349 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907340 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtr8\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-kube-api-access-rxtr8\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907362 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b201b8-05b4-4768-a9b0-bc762d71d79b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907437 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-token\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-envoy\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907497 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-socket\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907836 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907661 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-podinfo\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907836 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907723 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-data\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907836 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907786 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-credential-socket\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907836 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907802 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-certs\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.907993 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.907949 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-socket\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.908124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.908107 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b201b8-05b4-4768-a9b0-bc762d71d79b-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.908183 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.908136 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-credential-socket\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.909776 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.909757 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-envoy\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.909972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.909955 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-podinfo\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.915669 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.915639 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-token\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.915864 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.915844 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtr8\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-kube-api-access-rxtr8\") pod \"e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:58.958390 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:58.958370 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:42:59.259831 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:59.259802 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t"] Sep 10 17:42:59.266942 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:42:59.266915 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b201b8_05b4_4768_a9b0_bc762d71d79b.slice/crio-fb585ce4bb81f0e141f9e45822276a0cb0018db6f51890823b84bb3c542f45d6 WatchSource:0}: Error finding container fb585ce4bb81f0e141f9e45822276a0cb0018db6f51890823b84bb3c542f45d6: Status 404 returned error can't find the container with id fb585ce4bb81f0e141f9e45822276a0cb0018db6f51890823b84bb3c542f45d6 Sep 10 17:42:59.269282 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:59.269246 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:42:59.269374 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:59.269319 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:42:59.269374 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:42:59.269346 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:43:00.017980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:00.017946 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" event={"ID":"75b201b8-05b4-4768-a9b0-bc762d71d79b","Type":"ContainerStarted","Data":"3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b"} Sep 10 17:43:00.018441 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:00.017994 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" event={"ID":"75b201b8-05b4-4768-a9b0-bc762d71d79b","Type":"ContainerStarted","Data":"fb585ce4bb81f0e141f9e45822276a0cb0018db6f51890823b84bb3c542f45d6"} Sep 10 17:43:00.039784 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:00.039606 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" podStartSLOduration=2.039589096 podStartE2EDuration="2.039589096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:00.038394061 +0000 UTC m=+1779.136292432" watchObservedRunningTime="2026-09-10 17:43:00.039589096 +0000 UTC m=+1779.137487462" Sep 10 17:43:00.959321 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:00.959284 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:00.964314 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:00.964291 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:01.022987 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:01.022961 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:01.023774 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:01.023757 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:06.524613 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.524580 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp"] Sep 10 17:43:06.528409 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.528391 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:06.530543 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.530523 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8a34df\"" Sep 10 17:43:06.530662 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.530556 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8a34df-dockercfg-p4qzq\"" Sep 10 17:43:06.538091 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.538069 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp"] Sep 10 17:43:06.676454 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.676411 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svjsd\" (UniqueName: \"kubernetes.io/projected/691ecc08-ea4a-4bfd-a50c-572d8378265d-kube-api-access-svjsd\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp\" (UID: \"691ecc08-ea4a-4bfd-a50c-572d8378265d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:06.676615 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.676490 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ecc08-ea4a-4bfd-a50c-572d8378265d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp\" (UID: \"691ecc08-ea4a-4bfd-a50c-572d8378265d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:06.777259 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.777186 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ecc08-ea4a-4bfd-a50c-572d8378265d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp\" (UID: \"691ecc08-ea4a-4bfd-a50c-572d8378265d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:06.777379 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.777327 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svjsd\" (UniqueName: \"kubernetes.io/projected/691ecc08-ea4a-4bfd-a50c-572d8378265d-kube-api-access-svjsd\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp\" (UID: \"691ecc08-ea4a-4bfd-a50c-572d8378265d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:06.777885 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.777866 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ecc08-ea4a-4bfd-a50c-572d8378265d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp\" (UID: \"691ecc08-ea4a-4bfd-a50c-572d8378265d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:06.787565 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.787541 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svjsd\" (UniqueName: \"kubernetes.io/projected/691ecc08-ea4a-4bfd-a50c-572d8378265d-kube-api-access-svjsd\") pod \"payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp\" (UID: \"691ecc08-ea4a-4bfd-a50c-572d8378265d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:06.839609 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:06.839583 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:07.112880 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:07.112854 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp"] Sep 10 17:43:07.115285 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:43:07.115230 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691ecc08_ea4a_4bfd_a50c_572d8378265d.slice/crio-2d8086d87a7cced6fd849aa977922b7ce6013b04ca99d36663b4ef1e1d55d6fd WatchSource:0}: Error finding container 2d8086d87a7cced6fd849aa977922b7ce6013b04ca99d36663b4ef1e1d55d6fd: Status 404 returned error can't find the container with id 2d8086d87a7cced6fd849aa977922b7ce6013b04ca99d36663b4ef1e1d55d6fd Sep 10 17:43:08.058993 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:08.058956 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" event={"ID":"691ecc08-ea4a-4bfd-a50c-572d8378265d","Type":"ContainerStarted","Data":"ac05b69e45ecd050370a4ab8d614e68a1672fd90005891bcad7fc78e2da6a11d"} Sep 10 17:43:08.059450 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:08.059020 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" event={"ID":"691ecc08-ea4a-4bfd-a50c-572d8378265d","Type":"ContainerStarted","Data":"2d8086d87a7cced6fd849aa977922b7ce6013b04ca99d36663b4ef1e1d55d6fd"} Sep 10 17:43:08.059450 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:08.059066 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:08.079764 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:08.079721 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" podStartSLOduration=2.07970774 podStartE2EDuration="2.07970774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:43:08.078376904 +0000 UTC m=+1787.176275263" watchObservedRunningTime="2026-09-10 17:43:08.07970774 +0000 UTC m=+1787.177606099" Sep 10 17:43:19.066648 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:19.066620 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:21.421953 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:21.421930 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:43:21.427734 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:21.427717 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:43:22.130580 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:22.130547 2557 generic.go:356] "Generic (PLEG): container finished" podID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerID="3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b" exitCode=0 Sep 10 17:43:22.130731 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:22.130591 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" event={"ID":"75b201b8-05b4-4768-a9b0-bc762d71d79b","Type":"ContainerDied","Data":"3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b"} Sep 10 17:43:22.130923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:22.130910 2557 scope.go:117] "RemoveContainer" containerID="3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b" Sep 10 17:43:22.131915 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:22.131886 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:43:22.131969 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:22.131956 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:43:22.132009 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:22.131994 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 17:43:23.137228 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:23.137197 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" event={"ID":"75b201b8-05b4-4768-a9b0-bc762d71d79b","Type":"ContainerStarted","Data":"6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec"} Sep 10 17:43:23.959822 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:23.959788 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:23.960205 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:23.960123 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 17:43:23.960382 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:23.960358 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 17:43:24.959466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:24.959420 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 17:43:24.959956 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:24.959489 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 17:43:25.959665 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:25.959619 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 17:43:25.960184 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:25.959690 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 17:43:26.959211 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:26.959175 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 17:43:26.959503 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:26.959466 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 17:43:27.959009 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:27.958971 2557 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 10 17:43:27.959421 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:27.959063 2557 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 10 17:43:28.963122 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:28.963091 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:28.963521 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:28.963502 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:28.964007 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:28.963988 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:44.092187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.092124 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp"] Sep 10 17:43:44.092625 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.092392 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" podUID="691ecc08-ea4a-4bfd-a50c-572d8378265d" containerName="payload-pre-processing" containerID="cri-o://ac05b69e45ecd050370a4ab8d614e68a1672fd90005891bcad7fc78e2da6a11d" gracePeriod=30 Sep 10 17:43:44.250972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.250939 2557 generic.go:356] "Generic (PLEG): container finished" podID="691ecc08-ea4a-4bfd-a50c-572d8378265d" containerID="ac05b69e45ecd050370a4ab8d614e68a1672fd90005891bcad7fc78e2da6a11d" exitCode=0 Sep 10 17:43:44.251141 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.251021 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" event={"ID":"691ecc08-ea4a-4bfd-a50c-572d8378265d","Type":"ContainerDied","Data":"ac05b69e45ecd050370a4ab8d614e68a1672fd90005891bcad7fc78e2da6a11d"} Sep 10 17:43:44.340652 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.340631 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:44.400558 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.400496 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svjsd\" (UniqueName: \"kubernetes.io/projected/691ecc08-ea4a-4bfd-a50c-572d8378265d-kube-api-access-svjsd\") pod \"691ecc08-ea4a-4bfd-a50c-572d8378265d\" (UID: \"691ecc08-ea4a-4bfd-a50c-572d8378265d\") " Sep 10 17:43:44.400693 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.400582 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ecc08-ea4a-4bfd-a50c-572d8378265d-plugins-config-volume\") pod \"691ecc08-ea4a-4bfd-a50c-572d8378265d\" (UID: \"691ecc08-ea4a-4bfd-a50c-572d8378265d\") " Sep 10 17:43:44.400941 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.400918 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691ecc08-ea4a-4bfd-a50c-572d8378265d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "691ecc08-ea4a-4bfd-a50c-572d8378265d" (UID: "691ecc08-ea4a-4bfd-a50c-572d8378265d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:43:44.402562 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.402536 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691ecc08-ea4a-4bfd-a50c-572d8378265d-kube-api-access-svjsd" (OuterVolumeSpecName: "kube-api-access-svjsd") pod "691ecc08-ea4a-4bfd-a50c-572d8378265d" (UID: "691ecc08-ea4a-4bfd-a50c-572d8378265d"). InnerVolumeSpecName "kube-api-access-svjsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:43:44.502080 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.502019 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/691ecc08-ea4a-4bfd-a50c-572d8378265d-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:44.502080 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:44.502079 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svjsd\" (UniqueName: \"kubernetes.io/projected/691ecc08-ea4a-4bfd-a50c-572d8378265d-kube-api-access-svjsd\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:45.257685 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:45.257658 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" Sep 10 17:43:45.258134 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:45.257704 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp" event={"ID":"691ecc08-ea4a-4bfd-a50c-572d8378265d","Type":"ContainerDied","Data":"2d8086d87a7cced6fd849aa977922b7ce6013b04ca99d36663b4ef1e1d55d6fd"} Sep 10 17:43:45.258134 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:45.257771 2557 scope.go:117] "RemoveContainer" containerID="ac05b69e45ecd050370a4ab8d614e68a1672fd90005891bcad7fc78e2da6a11d" Sep 10 17:43:45.282971 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:45.282947 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp"] Sep 10 17:43:45.288436 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:45.288413 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8a34df-5b4fcb47d6-rd6dp"] Sep 10 17:43:45.386593 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:45.386569 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691ecc08-ea4a-4bfd-a50c-572d8378265d" path="/var/lib/kubelet/pods/691ecc08-ea4a-4bfd-a50c-572d8378265d/volumes" Sep 10 17:43:50.961908 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:50.961872 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t"] Sep 10 17:43:50.962454 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:50.962168 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" containerID="cri-o://6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec" gracePeriod=30 Sep 10 17:43:56.215549 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.215524 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:56.306967 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.306940 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-credential-socket\") pod \"75b201b8-05b4-4768-a9b0-bc762d71d79b\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " Sep 10 17:43:56.307127 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.306989 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-data\") pod \"75b201b8-05b4-4768-a9b0-bc762d71d79b\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " Sep 10 17:43:56.307127 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307028 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b201b8-05b4-4768-a9b0-bc762d71d79b-istiod-ca-cert\") pod \"75b201b8-05b4-4768-a9b0-bc762d71d79b\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " Sep 10 17:43:56.307127 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307081 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-podinfo\") pod \"75b201b8-05b4-4768-a9b0-bc762d71d79b\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " Sep 10 17:43:56.307127 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307107 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-token\") pod \"75b201b8-05b4-4768-a9b0-bc762d71d79b\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " Sep 10 17:43:56.307350 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307138 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxtr8\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-kube-api-access-rxtr8\") pod \"75b201b8-05b4-4768-a9b0-bc762d71d79b\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " Sep 10 17:43:56.307350 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307192 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-envoy\") pod \"75b201b8-05b4-4768-a9b0-bc762d71d79b\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " Sep 10 17:43:56.307350 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307197 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75b201b8-05b4-4768-a9b0-bc762d71d79b" (UID: "75b201b8-05b4-4768-a9b0-bc762d71d79b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:43:56.307350 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307230 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-certs\") pod \"75b201b8-05b4-4768-a9b0-bc762d71d79b\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " Sep 10 17:43:56.307350 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307266 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-data" (OuterVolumeSpecName: "istio-data") pod "75b201b8-05b4-4768-a9b0-bc762d71d79b" (UID: "75b201b8-05b4-4768-a9b0-bc762d71d79b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:43:56.307350 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307281 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-socket\") pod \"75b201b8-05b4-4768-a9b0-bc762d71d79b\" (UID: \"75b201b8-05b4-4768-a9b0-bc762d71d79b\") " Sep 10 17:43:56.307805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307554 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75b201b8-05b4-4768-a9b0-bc762d71d79b" (UID: "75b201b8-05b4-4768-a9b0-bc762d71d79b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:43:56.307805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307589 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:56.307805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307606 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:56.307805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307689 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75b201b8-05b4-4768-a9b0-bc762d71d79b" (UID: "75b201b8-05b4-4768-a9b0-bc762d71d79b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:43:56.308792 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.307949 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b201b8-05b4-4768-a9b0-bc762d71d79b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75b201b8-05b4-4768-a9b0-bc762d71d79b" (UID: "75b201b8-05b4-4768-a9b0-bc762d71d79b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:43:56.309875 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.309845 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-token" (OuterVolumeSpecName: "istio-token") pod "75b201b8-05b4-4768-a9b0-bc762d71d79b" (UID: "75b201b8-05b4-4768-a9b0-bc762d71d79b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:43:56.309980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.309902 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75b201b8-05b4-4768-a9b0-bc762d71d79b" (UID: "75b201b8-05b4-4768-a9b0-bc762d71d79b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:43:56.309980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.309908 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-kube-api-access-rxtr8" (OuterVolumeSpecName: "kube-api-access-rxtr8") pod "75b201b8-05b4-4768-a9b0-bc762d71d79b" (UID: "75b201b8-05b4-4768-a9b0-bc762d71d79b"). InnerVolumeSpecName "kube-api-access-rxtr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:43:56.310180 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.310157 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75b201b8-05b4-4768-a9b0-bc762d71d79b" (UID: "75b201b8-05b4-4768-a9b0-bc762d71d79b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:43:56.316554 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.316532 2557 generic.go:356] "Generic (PLEG): container finished" podID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerID="6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec" exitCode=0 Sep 10 17:43:56.316641 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.316570 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" event={"ID":"75b201b8-05b4-4768-a9b0-bc762d71d79b","Type":"ContainerDied","Data":"6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec"} Sep 10 17:43:56.316641 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.316597 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" Sep 10 17:43:56.316641 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.316622 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t" event={"ID":"75b201b8-05b4-4768-a9b0-bc762d71d79b","Type":"ContainerDied","Data":"fb585ce4bb81f0e141f9e45822276a0cb0018db6f51890823b84bb3c542f45d6"} Sep 10 17:43:56.316762 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.316644 2557 scope.go:117] "RemoveContainer" containerID="6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec" Sep 10 17:43:56.328517 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.328503 2557 scope.go:117] "RemoveContainer" containerID="3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b" Sep 10 17:43:56.336312 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.336295 2557 scope.go:117] "RemoveContainer" containerID="6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec" Sep 10 17:43:56.336600 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:43:56.336581 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec\": container with ID starting with 6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec not found: ID does not exist" containerID="6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec" Sep 10 17:43:56.336667 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.336608 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec"} err="failed to get container status \"6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec\": rpc error: code = NotFound desc = could not find container \"6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec\": container with ID starting with 6df8dd4a677564cee830da378726d6660de7f59858b1a34bf22f75146289b6ec not found: ID does not exist" Sep 10 17:43:56.336667 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.336624 2557 scope.go:117] "RemoveContainer" containerID="3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b" Sep 10 17:43:56.336849 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:43:56.336834 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b\": container with ID starting with 3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b not found: ID does not exist" containerID="3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b" Sep 10 17:43:56.336892 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.336852 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b"} err="failed to get container status \"3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b\": rpc error: code = NotFound desc = could not find container \"3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b\": container with ID starting with 3e96ce8a0fcebd714b9422d72187f865030af46bfb75e6d7f3eed7a3c410fa0b not found: ID does not exist" Sep 10 17:43:56.340320 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.340301 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t"] Sep 10 17:43:56.344157 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.344137 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8a34df-openshift-default-6bf55d9476-nhb7t"] Sep 10 17:43:56.408857 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.408832 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxtr8\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-kube-api-access-rxtr8\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:56.408857 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.408853 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:56.408986 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.408896 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:56.408986 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.408904 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b201b8-05b4-4768-a9b0-bc762d71d79b-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:56.408986 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.408912 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b201b8-05b4-4768-a9b0-bc762d71d79b-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:56.408986 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.408920 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:56.408986 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:56.408928 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b201b8-05b4-4768-a9b0-bc762d71d79b-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:43:57.385948 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:43:57.385915 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" path="/var/lib/kubelet/pods/75b201b8-05b4-4768-a9b0-bc762d71d79b/volumes" Sep 10 17:44:10.271000 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.270965 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8"] Sep 10 17:44:10.271404 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.271212 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" podUID="0cf629b6-dcd8-4875-b1c6-cb4c8de076c4" containerName="payload-processing" containerID="cri-o://b4259d24e8ef8dcf8ef78ef8726598c8fe30ca546d598d10277d8f5ec8f655e2" gracePeriod=30 Sep 10 17:44:10.280900 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.280871 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g"] Sep 10 17:44:10.281182 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.281101 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" podUID="0835426f-455a-4fa3-a63a-844a332bb0fc" containerName="payload-pre-processing" containerID="cri-o://ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5" gracePeriod=30 Sep 10 17:44:10.385838 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.385811 2557 generic.go:356] "Generic (PLEG): container finished" podID="0cf629b6-dcd8-4875-b1c6-cb4c8de076c4" containerID="b4259d24e8ef8dcf8ef78ef8726598c8fe30ca546d598d10277d8f5ec8f655e2" exitCode=0 Sep 10 17:44:10.385967 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.385876 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" event={"ID":"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4","Type":"ContainerDied","Data":"b4259d24e8ef8dcf8ef78ef8726598c8fe30ca546d598d10277d8f5ec8f655e2"} Sep 10 17:44:10.559245 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.559223 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:44:10.562474 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.562443 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:44:10.718385 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.718356 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwjwb\" (UniqueName: \"kubernetes.io/projected/0835426f-455a-4fa3-a63a-844a332bb0fc-kube-api-access-pwjwb\") pod \"0835426f-455a-4fa3-a63a-844a332bb0fc\" (UID: \"0835426f-455a-4fa3-a63a-844a332bb0fc\") " Sep 10 17:44:10.718558 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.718399 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2ccr\" (UniqueName: \"kubernetes.io/projected/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-kube-api-access-v2ccr\") pod \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\" (UID: \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\") " Sep 10 17:44:10.718558 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.718426 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835426f-455a-4fa3-a63a-844a332bb0fc-plugins-config-volume\") pod \"0835426f-455a-4fa3-a63a-844a332bb0fc\" (UID: \"0835426f-455a-4fa3-a63a-844a332bb0fc\") " Sep 10 17:44:10.718558 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.718479 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-plugins-config-volume\") pod \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\" (UID: \"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4\") " Sep 10 17:44:10.718822 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.718795 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0835426f-455a-4fa3-a63a-844a332bb0fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0835426f-455a-4fa3-a63a-844a332bb0fc" (UID: "0835426f-455a-4fa3-a63a-844a332bb0fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:44:10.718944 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.718866 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0cf629b6-dcd8-4875-b1c6-cb4c8de076c4" (UID: "0cf629b6-dcd8-4875-b1c6-cb4c8de076c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:44:10.720889 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.720868 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0835426f-455a-4fa3-a63a-844a332bb0fc-kube-api-access-pwjwb" (OuterVolumeSpecName: "kube-api-access-pwjwb") pod "0835426f-455a-4fa3-a63a-844a332bb0fc" (UID: "0835426f-455a-4fa3-a63a-844a332bb0fc"). InnerVolumeSpecName "kube-api-access-pwjwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:44:10.720971 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.720869 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-kube-api-access-v2ccr" (OuterVolumeSpecName: "kube-api-access-v2ccr") pod "0cf629b6-dcd8-4875-b1c6-cb4c8de076c4" (UID: "0cf629b6-dcd8-4875-b1c6-cb4c8de076c4"). InnerVolumeSpecName "kube-api-access-v2ccr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:44:10.819414 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.819356 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:44:10.819414 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.819379 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjwb\" (UniqueName: \"kubernetes.io/projected/0835426f-455a-4fa3-a63a-844a332bb0fc-kube-api-access-pwjwb\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:44:10.819414 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.819390 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ccr\" (UniqueName: \"kubernetes.io/projected/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4-kube-api-access-v2ccr\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:44:10.819414 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:10.819398 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835426f-455a-4fa3-a63a-844a332bb0fc-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:44:11.399433 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.399404 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" Sep 10 17:44:11.399900 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.399440 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8" event={"ID":"0cf629b6-dcd8-4875-b1c6-cb4c8de076c4","Type":"ContainerDied","Data":"59e4fb8e91473a39982eccaae24ed054702173b14c554d30f10e14d8d105704d"} Sep 10 17:44:11.399900 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.399497 2557 scope.go:117] "RemoveContainer" containerID="b4259d24e8ef8dcf8ef78ef8726598c8fe30ca546d598d10277d8f5ec8f655e2" Sep 10 17:44:11.401199 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.400955 2557 generic.go:356] "Generic (PLEG): container finished" podID="0835426f-455a-4fa3-a63a-844a332bb0fc" containerID="ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5" exitCode=0 Sep 10 17:44:11.401199 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.400991 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" event={"ID":"0835426f-455a-4fa3-a63a-844a332bb0fc","Type":"ContainerDied","Data":"ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5"} Sep 10 17:44:11.401199 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.401024 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" event={"ID":"0835426f-455a-4fa3-a63a-844a332bb0fc","Type":"ContainerDied","Data":"7c630802ed802627a2b22eb277659ebf47ea84663bd07c0d380e2db6ce0de1ab"} Sep 10 17:44:11.401199 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.401181 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g" Sep 10 17:44:11.410491 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.410445 2557 scope.go:117] "RemoveContainer" containerID="ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5" Sep 10 17:44:11.418102 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.418082 2557 scope.go:117] "RemoveContainer" containerID="ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5" Sep 10 17:44:11.418368 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:44:11.418350 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5\": container with ID starting with ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5 not found: ID does not exist" containerID="ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5" Sep 10 17:44:11.418432 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.418376 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5"} err="failed to get container status \"ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5\": rpc error: code = NotFound desc = could not find container \"ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5\": container with ID starting with ac4d85eed1aa82f2aef66a8e13003ca065550b62aa87cfdd8556e9e311a73bf5 not found: ID does not exist" Sep 10 17:44:11.419549 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.419531 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8"] Sep 10 17:44:11.423061 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.423009 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7858f0-78dcb5c965-xr4h8"] Sep 10 17:44:11.432263 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.432239 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g"] Sep 10 17:44:11.435461 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:11.435442 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7858f0-598f478d74-7nw5g"] Sep 10 17:44:13.386365 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:13.386332 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0835426f-455a-4fa3-a63a-844a332bb0fc" path="/var/lib/kubelet/pods/0835426f-455a-4fa3-a63a-844a332bb0fc/volumes" Sep 10 17:44:13.386701 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:13.386668 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf629b6-dcd8-4875-b1c6-cb4c8de076c4" path="/var/lib/kubelet/pods/0cf629b6-dcd8-4875-b1c6-cb4c8de076c4/volumes" Sep 10 17:44:59.048058 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:59.047964 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg"] Sep 10 17:44:59.048616 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:59.048252 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" containerID="cri-o://c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05" gracePeriod=30 Sep 10 17:44:59.485274 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:59.485192 2557 patch_prober.go:28] interesting pod/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 17:44:59.485414 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:44:59.485271 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 17:45:00.149648 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.149613 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr"] Sep 10 17:45:00.150006 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.149981 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0835426f-455a-4fa3-a63a-844a332bb0fc" containerName="payload-pre-processing" Sep 10 17:45:00.150006 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.149993 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0835426f-455a-4fa3-a63a-844a332bb0fc" containerName="payload-pre-processing" Sep 10 17:45:00.150006 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150004 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691ecc08-ea4a-4bfd-a50c-572d8378265d" containerName="payload-pre-processing" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150009 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="691ecc08-ea4a-4bfd-a50c-572d8378265d" containerName="payload-pre-processing" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150018 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150025 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150045 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150051 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150067 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf629b6-dcd8-4875-b1c6-cb4c8de076c4" containerName="payload-processing" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150072 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf629b6-dcd8-4875-b1c6-cb4c8de076c4" containerName="payload-processing" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150127 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0835426f-455a-4fa3-a63a-844a332bb0fc" containerName="payload-pre-processing" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150136 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf629b6-dcd8-4875-b1c6-cb4c8de076c4" containerName="payload-processing" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150141 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150146 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="691ecc08-ea4a-4bfd-a50c-572d8378265d" containerName="payload-pre-processing" Sep 10 17:45:00.150153 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.150155 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="75b201b8-05b4-4768-a9b0-bc762d71d79b" containerName="istio-proxy" Sep 10 17:45:00.153140 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.153119 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr" Sep 10 17:45:00.153499 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.153480 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq"] Sep 10 17:45:00.156174 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.156153 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wxddn\"" Sep 10 17:45:00.157255 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.157241 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq" Sep 10 17:45:00.161938 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.161917 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr"] Sep 10 17:45:00.166167 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.166146 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr" Sep 10 17:45:00.167680 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.167659 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq"] Sep 10 17:45:00.176117 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.176098 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq" Sep 10 17:45:00.444992 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.444966 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr"] Sep 10 17:45:00.446648 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:45:00.446620 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01138b4_1007_43a3_af9d_cffaf01a359f.slice/crio-e9e8a19f50c2d2a426488e890276ecf30fabcf412e6af71ddc798a6e0de41813 WatchSource:0}: Error finding container e9e8a19f50c2d2a426488e890276ecf30fabcf412e6af71ddc798a6e0de41813: Status 404 returned error can't find the container with id e9e8a19f50c2d2a426488e890276ecf30fabcf412e6af71ddc798a6e0de41813 Sep 10 17:45:00.448775 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.448757 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:45:00.466251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.466231 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq"] Sep 10 17:45:00.468148 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:45:00.468127 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a560afb_11d6_452c_8cc4_348f6639b8e6.slice/crio-369d62a84630a7e1ded8acce364d4c4d130e6ae8634879f3fb9ec48130b25521 WatchSource:0}: Error finding container 369d62a84630a7e1ded8acce364d4c4d130e6ae8634879f3fb9ec48130b25521: Status 404 returned error can't find the container with id 369d62a84630a7e1ded8acce364d4c4d130e6ae8634879f3fb9ec48130b25521 Sep 10 17:45:00.637101 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.637060 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq" event={"ID":"1a560afb-11d6-452c-8cc4-348f6639b8e6","Type":"ContainerStarted","Data":"369d62a84630a7e1ded8acce364d4c4d130e6ae8634879f3fb9ec48130b25521"} Sep 10 17:45:00.638272 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:00.638248 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr" event={"ID":"e01138b4-1007-43a3-af9d-cffaf01a359f","Type":"ContainerStarted","Data":"e9e8a19f50c2d2a426488e890276ecf30fabcf412e6af71ddc798a6e0de41813"} Sep 10 17:45:03.658788 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:03.658750 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq" event={"ID":"1a560afb-11d6-452c-8cc4-348f6639b8e6","Type":"ContainerStarted","Data":"19e2d2eef95a6f209d09b08fefee7076668824cfb61b8fc10e660baaa71e7935"} Sep 10 17:45:03.660285 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:03.660260 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr" event={"ID":"e01138b4-1007-43a3-af9d-cffaf01a359f","Type":"ContainerStarted","Data":"2ff313fd10ddc2ae0ccb40679f6fc47a58e575b3c3171ec8fa6376f90759f5c1"} Sep 10 17:45:03.679947 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:03.679907 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq" podStartSLOduration=1.162792922 podStartE2EDuration="3.679895343s" totalImagesPullingTime="2.517102421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:03.678800587 +0000 UTC m=+1902.776698948" watchObservedRunningTime="2026-09-10 17:45:03.679895343 +0000 UTC m=+1902.777793731" Sep 10 17:45:03.699724 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:03.699692 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr" podStartSLOduration=1.15737196 podStartE2EDuration="3.699680918s" totalImagesPullingTime="2.542308958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:03.699589054 +0000 UTC m=+1902.797487412" watchObservedRunningTime="2026-09-10 17:45:03.699680918 +0000 UTC m=+1902.797579277" Sep 10 17:45:04.296974 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.296950 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:45:04.357509 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357482 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-podinfo\") pod \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " Sep 10 17:45:04.357620 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357519 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-token\") pod \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " Sep 10 17:45:04.357620 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357568 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-socket\") pod \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " Sep 10 17:45:04.357620 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357588 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-envoy\") pod \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " Sep 10 17:45:04.357620 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357605 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-certs\") pod \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " Sep 10 17:45:04.357905 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357653 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istiod-ca-cert\") pod \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " Sep 10 17:45:04.357905 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357682 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-credential-socket\") pod \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " Sep 10 17:45:04.357905 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357736 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-data\") pod \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " Sep 10 17:45:04.357905 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357773 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjjlf\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-kube-api-access-fjjlf\") pod \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\" (UID: \"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf\") " Sep 10 17:45:04.357905 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357884 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" (UID: "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:04.358179 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.357927 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" (UID: "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:04.358179 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.358074 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" (UID: "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:04.358288 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.358173 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-data" (OuterVolumeSpecName: "istio-data") pod "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" (UID: "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:04.358288 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.358201 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:04.358288 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.358210 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" (UID: "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:45:04.358288 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.358224 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:04.358288 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.358260 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:04.359611 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.359594 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" (UID: "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:45:04.359722 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.359702 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" (UID: "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:04.360206 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.360189 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-kube-api-access-fjjlf" (OuterVolumeSpecName: "kube-api-access-fjjlf") pod "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" (UID: "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf"). InnerVolumeSpecName "kube-api-access-fjjlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:04.360258 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.360204 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-token" (OuterVolumeSpecName: "istio-token") pod "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" (UID: "03a49cb4-ec56-4581-9b0f-93c60d2ac6bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:04.459303 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.459247 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:04.459303 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.459280 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:04.459303 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.459290 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:04.459303 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.459298 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:04.459303 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.459306 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:04.459303 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.459313 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjjlf\" (UniqueName: \"kubernetes.io/projected/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf-kube-api-access-fjjlf\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:04.666998 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.666968 2557 generic.go:356] "Generic (PLEG): container finished" podID="1a560afb-11d6-452c-8cc4-348f6639b8e6" containerID="19e2d2eef95a6f209d09b08fefee7076668824cfb61b8fc10e660baaa71e7935" exitCode=0 Sep 10 17:45:04.667394 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.667072 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq" event={"ID":"1a560afb-11d6-452c-8cc4-348f6639b8e6","Type":"ContainerDied","Data":"19e2d2eef95a6f209d09b08fefee7076668824cfb61b8fc10e660baaa71e7935"} Sep 10 17:45:04.668390 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.668369 2557 generic.go:356] "Generic (PLEG): container finished" podID="e01138b4-1007-43a3-af9d-cffaf01a359f" containerID="2ff313fd10ddc2ae0ccb40679f6fc47a58e575b3c3171ec8fa6376f90759f5c1" exitCode=0 Sep 10 17:45:04.668493 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.668451 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr" event={"ID":"e01138b4-1007-43a3-af9d-cffaf01a359f","Type":"ContainerDied","Data":"2ff313fd10ddc2ae0ccb40679f6fc47a58e575b3c3171ec8fa6376f90759f5c1"} Sep 10 17:45:04.670065 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.670045 2557 generic.go:356] "Generic (PLEG): container finished" podID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerID="c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05" exitCode=0 Sep 10 17:45:04.670144 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.670085 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" Sep 10 17:45:04.670186 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.670085 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" event={"ID":"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf","Type":"ContainerDied","Data":"c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05"} Sep 10 17:45:04.670221 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.670209 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg" event={"ID":"03a49cb4-ec56-4581-9b0f-93c60d2ac6bf","Type":"ContainerDied","Data":"038bcf2c4b1512d2bdea4d93846ae2beae6f2905515441df98a1f701ee545f1d"} Sep 10 17:45:04.670263 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.670245 2557 scope.go:117] "RemoveContainer" containerID="c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05" Sep 10 17:45:04.681719 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.681660 2557 scope.go:117] "RemoveContainer" containerID="35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d" Sep 10 17:45:04.689317 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.689301 2557 scope.go:117] "RemoveContainer" containerID="c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05" Sep 10 17:45:04.689571 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:45:04.689552 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05\": container with ID starting with c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05 not found: ID does not exist" containerID="c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05" Sep 10 17:45:04.689639 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.689581 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05"} err="failed to get container status \"c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05\": rpc error: code = NotFound desc = could not find container \"c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05\": container with ID starting with c0439bdb564c22eeded386e8041ab8fa648e677788abd1cef266bd575c033f05 not found: ID does not exist" Sep 10 17:45:04.689639 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.689606 2557 scope.go:117] "RemoveContainer" containerID="35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d" Sep 10 17:45:04.689876 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:45:04.689859 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d\": container with ID starting with 35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d not found: ID does not exist" containerID="35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d" Sep 10 17:45:04.689920 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.689881 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d"} err="failed to get container status \"35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d\": rpc error: code = NotFound desc = could not find container \"35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d\": container with ID starting with 35177f04d71d974f1e6cd6e45e65c06cbde3091a06ffc738bc32f15b75793f5d not found: ID does not exist" Sep 10 17:45:04.694186 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.694166 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg"] Sep 10 17:45:04.701289 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:04.701270 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7858f0-openshift-default-5fb59bcd6b-cqkdg"] Sep 10 17:45:05.386182 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:05.386149 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" path="/var/lib/kubelet/pods/03a49cb4-ec56-4581-9b0f-93c60d2ac6bf/volumes" Sep 10 17:45:05.808139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:05.808115 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq" Sep 10 17:45:05.837992 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:05.837973 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr" Sep 10 17:45:06.682297 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:06.682268 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq" Sep 10 17:45:06.682471 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:06.682272 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq" event={"ID":"1a560afb-11d6-452c-8cc4-348f6639b8e6","Type":"ContainerDied","Data":"369d62a84630a7e1ded8acce364d4c4d130e6ae8634879f3fb9ec48130b25521"} Sep 10 17:45:06.682471 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:06.682384 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369d62a84630a7e1ded8acce364d4c4d130e6ae8634879f3fb9ec48130b25521" Sep 10 17:45:06.683551 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:06.683534 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr" Sep 10 17:45:06.683659 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:06.683547 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-g2xzr" event={"ID":"e01138b4-1007-43a3-af9d-cffaf01a359f","Type":"ContainerDied","Data":"e9e8a19f50c2d2a426488e890276ecf30fabcf412e6af71ddc798a6e0de41813"} Sep 10 17:45:06.683659 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:06.683582 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e8a19f50c2d2a426488e890276ecf30fabcf412e6af71ddc798a6e0de41813" Sep 10 17:45:10.005857 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.005824 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw"] Sep 10 17:45:10.006247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.006083 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" podUID="0690813f-5d44-4e99-8783-7764efb4bbfe" containerName="payload-processing" containerID="cri-o://4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed" gracePeriod=30 Sep 10 17:45:10.011302 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.011273 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc"] Sep 10 17:45:10.011505 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.011477 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" podUID="12bbd43f-40bb-464e-bf38-44590b92069a" containerName="payload-pre-processing" containerID="cri-o://a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a" gracePeriod=30 Sep 10 17:45:10.300945 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.300921 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:45:10.303549 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.303532 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:45:10.403897 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.403869 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt2mp\" (UniqueName: \"kubernetes.io/projected/12bbd43f-40bb-464e-bf38-44590b92069a-kube-api-access-mt2mp\") pod \"12bbd43f-40bb-464e-bf38-44590b92069a\" (UID: \"12bbd43f-40bb-464e-bf38-44590b92069a\") " Sep 10 17:45:10.404079 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.403991 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bbd43f-40bb-464e-bf38-44590b92069a-plugins-config-volume\") pod \"12bbd43f-40bb-464e-bf38-44590b92069a\" (UID: \"12bbd43f-40bb-464e-bf38-44590b92069a\") " Sep 10 17:45:10.404079 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.404016 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n55pb\" (UniqueName: \"kubernetes.io/projected/0690813f-5d44-4e99-8783-7764efb4bbfe-kube-api-access-n55pb\") pod \"0690813f-5d44-4e99-8783-7764efb4bbfe\" (UID: \"0690813f-5d44-4e99-8783-7764efb4bbfe\") " Sep 10 17:45:10.404079 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.404056 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690813f-5d44-4e99-8783-7764efb4bbfe-plugins-config-volume\") pod \"0690813f-5d44-4e99-8783-7764efb4bbfe\" (UID: \"0690813f-5d44-4e99-8783-7764efb4bbfe\") " Sep 10 17:45:10.404398 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.404363 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12bbd43f-40bb-464e-bf38-44590b92069a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12bbd43f-40bb-464e-bf38-44590b92069a" (UID: "12bbd43f-40bb-464e-bf38-44590b92069a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:45:10.404522 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.404409 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0690813f-5d44-4e99-8783-7764efb4bbfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0690813f-5d44-4e99-8783-7764efb4bbfe" (UID: "0690813f-5d44-4e99-8783-7764efb4bbfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:45:10.405915 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.405889 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bbd43f-40bb-464e-bf38-44590b92069a-kube-api-access-mt2mp" (OuterVolumeSpecName: "kube-api-access-mt2mp") pod "12bbd43f-40bb-464e-bf38-44590b92069a" (UID: "12bbd43f-40bb-464e-bf38-44590b92069a"). InnerVolumeSpecName "kube-api-access-mt2mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:10.406001 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.405929 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0690813f-5d44-4e99-8783-7764efb4bbfe-kube-api-access-n55pb" (OuterVolumeSpecName: "kube-api-access-n55pb") pod "0690813f-5d44-4e99-8783-7764efb4bbfe" (UID: "0690813f-5d44-4e99-8783-7764efb4bbfe"). InnerVolumeSpecName "kube-api-access-n55pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:10.504840 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.504813 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12bbd43f-40bb-464e-bf38-44590b92069a-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:10.504840 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.504833 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n55pb\" (UniqueName: \"kubernetes.io/projected/0690813f-5d44-4e99-8783-7764efb4bbfe-kube-api-access-n55pb\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:10.504840 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.504843 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0690813f-5d44-4e99-8783-7764efb4bbfe-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:10.505018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.504851 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt2mp\" (UniqueName: \"kubernetes.io/projected/12bbd43f-40bb-464e-bf38-44590b92069a-kube-api-access-mt2mp\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:10.704719 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.704626 2557 generic.go:356] "Generic (PLEG): container finished" podID="0690813f-5d44-4e99-8783-7764efb4bbfe" containerID="4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed" exitCode=0 Sep 10 17:45:10.704881 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.704773 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" Sep 10 17:45:10.704881 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.704786 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" event={"ID":"0690813f-5d44-4e99-8783-7764efb4bbfe","Type":"ContainerDied","Data":"4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed"} Sep 10 17:45:10.704881 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.704842 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw" event={"ID":"0690813f-5d44-4e99-8783-7764efb4bbfe","Type":"ContainerDied","Data":"0712d55652aba894d658441350d4e55dc615b187cf6079779f3f751b5b1cb231"} Sep 10 17:45:10.704881 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.704871 2557 scope.go:117] "RemoveContainer" containerID="4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed" Sep 10 17:45:10.709192 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.709159 2557 generic.go:356] "Generic (PLEG): container finished" podID="12bbd43f-40bb-464e-bf38-44590b92069a" containerID="a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a" exitCode=0 Sep 10 17:45:10.709317 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.709272 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" event={"ID":"12bbd43f-40bb-464e-bf38-44590b92069a","Type":"ContainerDied","Data":"a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a"} Sep 10 17:45:10.709386 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.709316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" event={"ID":"12bbd43f-40bb-464e-bf38-44590b92069a","Type":"ContainerDied","Data":"bafe42ab6186e2845eb4528fb657734c4f030829e6f2eda6005595f1fb03a979"} Sep 10 17:45:10.709442 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.709389 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc" Sep 10 17:45:10.717956 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.717936 2557 scope.go:117] "RemoveContainer" containerID="4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed" Sep 10 17:45:10.718243 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:45:10.718220 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed\": container with ID starting with 4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed not found: ID does not exist" containerID="4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed" Sep 10 17:45:10.718325 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.718249 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed"} err="failed to get container status \"4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed\": rpc error: code = NotFound desc = could not find container \"4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed\": container with ID starting with 4c8170a72cfac58ced6e92bb6a72e4233c22e5954d28d723ba22fdc7c10895ed not found: ID does not exist" Sep 10 17:45:10.718325 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.718265 2557 scope.go:117] "RemoveContainer" containerID="a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a" Sep 10 17:45:10.726212 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.726196 2557 scope.go:117] "RemoveContainer" containerID="a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a" Sep 10 17:45:10.726452 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:45:10.726436 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a\": container with ID starting with a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a not found: ID does not exist" containerID="a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a" Sep 10 17:45:10.726496 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.726459 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a"} err="failed to get container status \"a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a\": rpc error: code = NotFound desc = could not find container \"a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a\": container with ID starting with a70b85247f5d269af6edcaef9f5e05fb350084af5c0489fae88ac811e336a20a not found: ID does not exist" Sep 10 17:45:10.730445 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.730423 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw"] Sep 10 17:45:10.735361 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.735342 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fdb1bd-88dc9bdb5-hw4nw"] Sep 10 17:45:10.746418 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.746396 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc"] Sep 10 17:45:10.748954 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:10.748935 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fdb1bd-65f6bc4cdb-jmxkc"] Sep 10 17:45:11.386102 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:11.386071 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0690813f-5d44-4e99-8783-7764efb4bbfe" path="/var/lib/kubelet/pods/0690813f-5d44-4e99-8783-7764efb4bbfe/volumes" Sep 10 17:45:11.386447 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:11.386431 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12bbd43f-40bb-464e-bf38-44590b92069a" path="/var/lib/kubelet/pods/12bbd43f-40bb-464e-bf38-44590b92069a/volumes" Sep 10 17:45:16.788491 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:16.788455 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd"] Sep 10 17:45:16.789080 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:16.788698 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" containerID="cri-o://14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1" gracePeriod=30 Sep 10 17:45:22.029630 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.029608 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:45:22.205423 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205344 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f9e963-90ce-43c3-a35c-ce9e164117ae-istiod-ca-cert\") pod \"64f9e963-90ce-43c3-a35c-ce9e164117ae\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " Sep 10 17:45:22.205423 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205381 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-data\") pod \"64f9e963-90ce-43c3-a35c-ce9e164117ae\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " Sep 10 17:45:22.205423 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205409 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-envoy\") pod \"64f9e963-90ce-43c3-a35c-ce9e164117ae\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " Sep 10 17:45:22.205675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205437 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-token\") pod \"64f9e963-90ce-43c3-a35c-ce9e164117ae\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " Sep 10 17:45:22.205675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205497 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-socket\") pod \"64f9e963-90ce-43c3-a35c-ce9e164117ae\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " Sep 10 17:45:22.205675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205523 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-podinfo\") pod \"64f9e963-90ce-43c3-a35c-ce9e164117ae\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " Sep 10 17:45:22.205675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205545 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-credential-socket\") pod \"64f9e963-90ce-43c3-a35c-ce9e164117ae\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " Sep 10 17:45:22.205675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205592 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48cxn\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-kube-api-access-48cxn\") pod \"64f9e963-90ce-43c3-a35c-ce9e164117ae\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " Sep 10 17:45:22.205675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205625 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-certs\") pod \"64f9e963-90ce-43c3-a35c-ce9e164117ae\" (UID: \"64f9e963-90ce-43c3-a35c-ce9e164117ae\") " Sep 10 17:45:22.205980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205689 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "64f9e963-90ce-43c3-a35c-ce9e164117ae" (UID: "64f9e963-90ce-43c3-a35c-ce9e164117ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:22.205980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205732 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f9e963-90ce-43c3-a35c-ce9e164117ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64f9e963-90ce-43c3-a35c-ce9e164117ae" (UID: "64f9e963-90ce-43c3-a35c-ce9e164117ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:45:22.205980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205775 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64f9e963-90ce-43c3-a35c-ce9e164117ae" (UID: "64f9e963-90ce-43c3-a35c-ce9e164117ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:22.205980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205866 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64f9e963-90ce-43c3-a35c-ce9e164117ae" (UID: "64f9e963-90ce-43c3-a35c-ce9e164117ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:22.205980 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.205966 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64f9e963-90ce-43c3-a35c-ce9e164117ae" (UID: "64f9e963-90ce-43c3-a35c-ce9e164117ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:22.206247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.206063 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f9e963-90ce-43c3-a35c-ce9e164117ae-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:22.206247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.206080 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:22.206247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.206093 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:22.206247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.206107 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:22.206247 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.206121 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:22.207567 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.207541 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64f9e963-90ce-43c3-a35c-ce9e164117ae" (UID: "64f9e963-90ce-43c3-a35c-ce9e164117ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:22.207670 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.207568 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "64f9e963-90ce-43c3-a35c-ce9e164117ae" (UID: "64f9e963-90ce-43c3-a35c-ce9e164117ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:22.207670 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.207623 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-kube-api-access-48cxn" (OuterVolumeSpecName: "kube-api-access-48cxn") pod "64f9e963-90ce-43c3-a35c-ce9e164117ae" (UID: "64f9e963-90ce-43c3-a35c-ce9e164117ae"). InnerVolumeSpecName "kube-api-access-48cxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:22.207783 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.207769 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64f9e963-90ce-43c3-a35c-ce9e164117ae" (UID: "64f9e963-90ce-43c3-a35c-ce9e164117ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:45:22.306697 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.306664 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:22.306697 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.306686 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48cxn\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-kube-api-access-48cxn\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:22.306818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.306701 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:22.306818 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.306714 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f9e963-90ce-43c3-a35c-ce9e164117ae-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:22.768995 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.768965 2557 generic.go:356] "Generic (PLEG): container finished" podID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerID="14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1" exitCode=0 Sep 10 17:45:22.769139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.769005 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" event={"ID":"64f9e963-90ce-43c3-a35c-ce9e164117ae","Type":"ContainerDied","Data":"14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1"} Sep 10 17:45:22.769139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.769019 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" Sep 10 17:45:22.769139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.769075 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd" event={"ID":"64f9e963-90ce-43c3-a35c-ce9e164117ae","Type":"ContainerDied","Data":"1fe08005dae1d7d0ec5c5d71a45e10b80aa142cd7980f5257dfc8da65ad1e40c"} Sep 10 17:45:22.769139 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.769110 2557 scope.go:117] "RemoveContainer" containerID="14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1" Sep 10 17:45:22.780599 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.780584 2557 scope.go:117] "RemoveContainer" containerID="3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208" Sep 10 17:45:22.787958 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.787937 2557 scope.go:117] "RemoveContainer" containerID="14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1" Sep 10 17:45:22.788233 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:45:22.788215 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1\": container with ID starting with 14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1 not found: ID does not exist" containerID="14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1" Sep 10 17:45:22.788291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.788240 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1"} err="failed to get container status \"14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1\": rpc error: code = NotFound desc = could not find container \"14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1\": container with ID starting with 14c8be09697c0f96090f5f80b4eca80cc8dbda497a8e63025d0568a3424b4ee1 not found: ID does not exist" Sep 10 17:45:22.788291 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.788256 2557 scope.go:117] "RemoveContainer" containerID="3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208" Sep 10 17:45:22.788472 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:45:22.788456 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208\": container with ID starting with 3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208 not found: ID does not exist" containerID="3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208" Sep 10 17:45:22.788520 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.788476 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208"} err="failed to get container status \"3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208\": rpc error: code = NotFound desc = could not find container \"3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208\": container with ID starting with 3de8a48f89c6d0aa2688a59da310eb4c4f156f07f2c4a3e1499b421289cf8208 not found: ID does not exist" Sep 10 17:45:22.791235 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.791214 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd"] Sep 10 17:45:22.797165 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:22.797145 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fdb1bd-openshift-default-797f5f48d6-5rvtd"] Sep 10 17:45:23.386298 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:23.386267 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" path="/var/lib/kubelet/pods/64f9e963-90ce-43c3-a35c-ce9e164117ae/volumes" Sep 10 17:45:35.652902 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:35.652870 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx"] Sep 10 17:45:35.653373 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:35.653127 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" containerID="cri-o://65f37d1b50cfe6aa8baa30f218207d24a1497c2a6bc19b35999a93191853c966" gracePeriod=30 Sep 10 17:45:40.856870 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.856839 2557 generic.go:356] "Generic (PLEG): container finished" podID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerID="65f37d1b50cfe6aa8baa30f218207d24a1497c2a6bc19b35999a93191853c966" exitCode=0 Sep 10 17:45:40.857258 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.856911 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" event={"ID":"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150","Type":"ContainerDied","Data":"65f37d1b50cfe6aa8baa30f218207d24a1497c2a6bc19b35999a93191853c966"} Sep 10 17:45:40.857258 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.856971 2557 scope.go:117] "RemoveContainer" containerID="01888e6cd46718260d2242c0a66c185d43d26d652451ae5a43fb630d9dea942d" Sep 10 17:45:40.897583 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.897565 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:45:40.960236 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960174 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-data\") pod \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " Sep 10 17:45:40.960236 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960219 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-certs\") pod \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " Sep 10 17:45:40.960399 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960250 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-token\") pod \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " Sep 10 17:45:40.960399 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960305 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4fjl\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-kube-api-access-w4fjl\") pod \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " Sep 10 17:45:40.960399 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960335 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-podinfo\") pod \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " Sep 10 17:45:40.960563 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960454 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" (UID: "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:40.960563 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960468 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-data" (OuterVolumeSpecName: "istio-data") pod "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" (UID: "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:40.960563 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960478 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-envoy\") pod \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " Sep 10 17:45:40.960563 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960522 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istiod-ca-cert\") pod \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " Sep 10 17:45:40.960563 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960557 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-credential-socket\") pod \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " Sep 10 17:45:40.960840 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960573 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-socket\") pod \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\" (UID: \"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150\") " Sep 10 17:45:40.960898 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960847 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" (UID: "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:45:40.960898 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960858 2557 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-data\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:40.960898 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960878 2557 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:40.961064 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960948 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" (UID: "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:40.961064 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.960960 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" (UID: "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:40.962408 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.962383 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" (UID: "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 17:45:40.962408 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.962395 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-token" (OuterVolumeSpecName: "istio-token") pod "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" (UID: "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:40.962757 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.962732 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" (UID: "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:45:40.962814 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:40.962735 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-kube-api-access-w4fjl" (OuterVolumeSpecName: "kube-api-access-w4fjl") pod "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" (UID: "7eefff4e-5d29-4f2c-af9a-5c1a1ec04150"). InnerVolumeSpecName "kube-api-access-w4fjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:41.061441 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.061416 2557 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-credential-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:41.061441 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.061440 2557 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-workload-socket\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:41.061600 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.061449 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-token\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:41.061600 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.061458 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4fjl\" (UniqueName: \"kubernetes.io/projected/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-kube-api-access-w4fjl\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:41.061600 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.061466 2557 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-podinfo\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:41.061600 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.061477 2557 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istio-envoy\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:41.061600 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.061488 2557 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150-istiod-ca-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:41.862920 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.862886 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" event={"ID":"7eefff4e-5d29-4f2c-af9a-5c1a1ec04150","Type":"ContainerDied","Data":"12d68dc121630922d8354ea25f456a7b5ccd36d2f22b64343b78f865e2484a00"} Sep 10 17:45:41.863400 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.862931 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx" Sep 10 17:45:41.863400 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.862963 2557 scope.go:117] "RemoveContainer" containerID="65f37d1b50cfe6aa8baa30f218207d24a1497c2a6bc19b35999a93191853c966" Sep 10 17:45:41.880104 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.880079 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx"] Sep 10 17:45:41.886149 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:41.886132 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3c30a2-openshift-default-697bfc554d-xfmsx"] Sep 10 17:45:43.386602 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:43.386566 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" path="/var/lib/kubelet/pods/7eefff4e-5d29-4f2c-af9a-5c1a1ec04150/volumes" Sep 10 17:45:46.548987 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.548949 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq"] Sep 10 17:45:46.552079 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.552017 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c1fb61-29817705-v98mq"] Sep 10 17:45:46.589044 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.589005 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp"] Sep 10 17:45:46.589280 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.589238 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" podUID="4d6b97eb-c464-4c62-865a-064f5bb9428d" containerName="payload-pre-processing" containerID="cri-o://9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd" gracePeriod=30 Sep 10 17:45:46.848271 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.848240 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:45:46.890982 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.890937 2557 generic.go:356] "Generic (PLEG): container finished" podID="4d6b97eb-c464-4c62-865a-064f5bb9428d" containerID="9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd" exitCode=0 Sep 10 17:45:46.891161 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.891063 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" event={"ID":"4d6b97eb-c464-4c62-865a-064f5bb9428d","Type":"ContainerDied","Data":"9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd"} Sep 10 17:45:46.891161 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.891070 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" Sep 10 17:45:46.891161 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.891116 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp" event={"ID":"4d6b97eb-c464-4c62-865a-064f5bb9428d","Type":"ContainerDied","Data":"a69915f412ff0d12e49170680c8ebfb16552961fd02f1398aeb767c53606d15e"} Sep 10 17:45:46.891161 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.891146 2557 scope.go:117] "RemoveContainer" containerID="9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd" Sep 10 17:45:46.903230 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.903208 2557 scope.go:117] "RemoveContainer" containerID="9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd" Sep 10 17:45:46.903518 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:45:46.903499 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd\": container with ID starting with 9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd not found: ID does not exist" containerID="9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd" Sep 10 17:45:46.903591 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.903531 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd"} err="failed to get container status \"9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd\": rpc error: code = NotFound desc = could not find container \"9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd\": container with ID starting with 9f737f5e76adb4763643518b6f09e1df9033471cad091fb542a7f6703cc27cbd not found: ID does not exist" Sep 10 17:45:46.907861 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.907845 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxnmd\" (UniqueName: \"kubernetes.io/projected/4d6b97eb-c464-4c62-865a-064f5bb9428d-kube-api-access-lxnmd\") pod \"4d6b97eb-c464-4c62-865a-064f5bb9428d\" (UID: \"4d6b97eb-c464-4c62-865a-064f5bb9428d\") " Sep 10 17:45:46.907923 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.907873 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6b97eb-c464-4c62-865a-064f5bb9428d-plugins-config-volume\") pod \"4d6b97eb-c464-4c62-865a-064f5bb9428d\" (UID: \"4d6b97eb-c464-4c62-865a-064f5bb9428d\") " Sep 10 17:45:46.908256 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.908233 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6b97eb-c464-4c62-865a-064f5bb9428d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d6b97eb-c464-4c62-865a-064f5bb9428d" (UID: "4d6b97eb-c464-4c62-865a-064f5bb9428d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:45:46.909858 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:46.909834 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6b97eb-c464-4c62-865a-064f5bb9428d-kube-api-access-lxnmd" (OuterVolumeSpecName: "kube-api-access-lxnmd") pod "4d6b97eb-c464-4c62-865a-064f5bb9428d" (UID: "4d6b97eb-c464-4c62-865a-064f5bb9428d"). InnerVolumeSpecName "kube-api-access-lxnmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:47.009485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:47.009450 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxnmd\" (UniqueName: \"kubernetes.io/projected/4d6b97eb-c464-4c62-865a-064f5bb9428d-kube-api-access-lxnmd\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:47.009485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:47.009478 2557 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d6b97eb-c464-4c62-865a-064f5bb9428d-plugins-config-volume\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:47.213278 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:47.213248 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp"] Sep 10 17:45:47.223363 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:47.223338 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c1fb61-6dcbd87466-2vxdp"] Sep 10 17:45:47.391932 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:47.391895 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a560afb-11d6-452c-8cc4-348f6639b8e6" path="/var/lib/kubelet/pods/1a560afb-11d6-452c-8cc4-348f6639b8e6/volumes" Sep 10 17:45:47.392251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:47.392238 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d6b97eb-c464-4c62-865a-064f5bb9428d" path="/var/lib/kubelet/pods/4d6b97eb-c464-4c62-865a-064f5bb9428d/volumes" Sep 10 17:45:55.947956 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:55.947923 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-4j5qz"] Sep 10 17:45:55.948438 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:55.948166 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" podUID="ce96cb77-d52b-462c-82d3-721b3f05f659" containerName="manager" containerID="cri-o://e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2" gracePeriod=30 Sep 10 17:45:56.190449 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.190426 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:45:56.286102 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.286073 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-metrics-certs\") pod \"ce96cb77-d52b-462c-82d3-721b3f05f659\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " Sep 10 17:45:56.286261 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.286118 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdpkx\" (UniqueName: \"kubernetes.io/projected/ce96cb77-d52b-462c-82d3-721b3f05f659-kube-api-access-xdpkx\") pod \"ce96cb77-d52b-462c-82d3-721b3f05f659\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " Sep 10 17:45:56.286261 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.286181 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-webhook-cert\") pod \"ce96cb77-d52b-462c-82d3-721b3f05f659\" (UID: \"ce96cb77-d52b-462c-82d3-721b3f05f659\") " Sep 10 17:45:56.288151 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.288125 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ce96cb77-d52b-462c-82d3-721b3f05f659" (UID: "ce96cb77-d52b-462c-82d3-721b3f05f659"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:45:56.288277 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.288149 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ce96cb77-d52b-462c-82d3-721b3f05f659" (UID: "ce96cb77-d52b-462c-82d3-721b3f05f659"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:45:56.288277 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.288264 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce96cb77-d52b-462c-82d3-721b3f05f659-kube-api-access-xdpkx" (OuterVolumeSpecName: "kube-api-access-xdpkx") pod "ce96cb77-d52b-462c-82d3-721b3f05f659" (UID: "ce96cb77-d52b-462c-82d3-721b3f05f659"). InnerVolumeSpecName "kube-api-access-xdpkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:45:56.387594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.387567 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-webhook-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:56.387594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.387591 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce96cb77-d52b-462c-82d3-721b3f05f659-metrics-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:56.387594 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.387599 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdpkx\" (UniqueName: \"kubernetes.io/projected/ce96cb77-d52b-462c-82d3-721b3f05f659-kube-api-access-xdpkx\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:45:56.938868 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.938833 2557 generic.go:356] "Generic (PLEG): container finished" podID="ce96cb77-d52b-462c-82d3-721b3f05f659" containerID="e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2" exitCode=0 Sep 10 17:45:56.939051 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.938872 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" Sep 10 17:45:56.939051 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.938885 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" event={"ID":"ce96cb77-d52b-462c-82d3-721b3f05f659","Type":"ContainerDied","Data":"e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2"} Sep 10 17:45:56.939051 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.938930 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-4j5qz" event={"ID":"ce96cb77-d52b-462c-82d3-721b3f05f659","Type":"ContainerDied","Data":"f5d8cb56bdbcdb22b576afdc5f7b13b1bbe670af161d15b1e30d2f65893aaae3"} Sep 10 17:45:56.939051 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.938953 2557 scope.go:117] "RemoveContainer" containerID="e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2" Sep 10 17:45:56.948245 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.948228 2557 scope.go:117] "RemoveContainer" containerID="e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2" Sep 10 17:45:56.948543 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:45:56.948527 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2\": container with ID starting with e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2 not found: ID does not exist" containerID="e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2" Sep 10 17:45:56.948585 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.948550 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2"} err="failed to get container status \"e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2\": rpc error: code = NotFound desc = could not find container \"e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2\": container with ID starting with e0c409b4e11e8f27780d1bd86185f833027519fa554df4575f319e9f31c901b2 not found: ID does not exist" Sep 10 17:45:56.958134 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.958113 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-4j5qz"] Sep 10 17:45:56.962237 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:56.962217 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-4j5qz"] Sep 10 17:45:57.385675 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:45:57.385642 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce96cb77-d52b-462c-82d3-721b3f05f659" path="/var/lib/kubelet/pods/ce96cb77-d52b-462c-82d3-721b3f05f659/volumes" Sep 10 17:46:57.800522 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800491 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-vt9r7"] Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800814 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6b97eb-c464-4c62-865a-064f5bb9428d" containerName="payload-pre-processing" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800828 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6b97eb-c464-4c62-865a-064f5bb9428d" containerName="payload-pre-processing" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800838 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800844 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800851 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800856 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800863 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800868 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800877 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800883 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800891 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0690813f-5d44-4e99-8783-7764efb4bbfe" containerName="payload-processing" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800900 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="0690813f-5d44-4e99-8783-7764efb4bbfe" containerName="payload-processing" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800907 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce96cb77-d52b-462c-82d3-721b3f05f659" containerName="manager" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800911 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce96cb77-d52b-462c-82d3-721b3f05f659" containerName="manager" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800931 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a560afb-11d6-452c-8cc4-348f6639b8e6" containerName="cleanup" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800936 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a560afb-11d6-452c-8cc4-348f6639b8e6" containerName="cleanup" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800948 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12bbd43f-40bb-464e-bf38-44590b92069a" containerName="payload-pre-processing" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800953 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="12bbd43f-40bb-464e-bf38-44590b92069a" containerName="payload-pre-processing" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800968 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01138b4-1007-43a3-af9d-cffaf01a359f" containerName="cleanup" Sep 10 17:46:57.800972 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.800977 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01138b4-1007-43a3-af9d-cffaf01a359f" containerName="cleanup" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801023 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801029 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce96cb77-d52b-462c-82d3-721b3f05f659" containerName="manager" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801050 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801059 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801066 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="12bbd43f-40bb-464e-bf38-44590b92069a" containerName="payload-pre-processing" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801071 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a560afb-11d6-452c-8cc4-348f6639b8e6" containerName="cleanup" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801076 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801081 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eefff4e-5d29-4f2c-af9a-5c1a1ec04150" containerName="istio-proxy" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801085 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01138b4-1007-43a3-af9d-cffaf01a359f" containerName="cleanup" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801089 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="0690813f-5d44-4e99-8783-7764efb4bbfe" containerName="payload-processing" Sep 10 17:46:57.801610 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.801094 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d6b97eb-c464-4c62-865a-064f5bb9428d" containerName="payload-pre-processing" Sep 10 17:46:57.803089 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.803069 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:57.805497 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.805471 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 17:46:57.806094 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.806077 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 17:46:57.806199 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.806137 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8xsv\"" Sep 10 17:46:57.806265 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.806221 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 17:46:57.813742 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.813722 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-vt9r7"] Sep 10 17:46:57.839837 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.839812 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-webhook-cert\") pod \"maas-controller-65bb9fc5b4-vt9r7\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:57.839926 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.839855 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5f7\" (UniqueName: \"kubernetes.io/projected/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-kube-api-access-5x5f7\") pod \"maas-controller-65bb9fc5b4-vt9r7\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:57.839926 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.839877 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-metrics-certs\") pod \"maas-controller-65bb9fc5b4-vt9r7\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:57.940432 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.940407 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-webhook-cert\") pod \"maas-controller-65bb9fc5b4-vt9r7\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:57.940537 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.940449 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5f7\" (UniqueName: \"kubernetes.io/projected/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-kube-api-access-5x5f7\") pod \"maas-controller-65bb9fc5b4-vt9r7\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:57.940537 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.940487 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-metrics-certs\") pod \"maas-controller-65bb9fc5b4-vt9r7\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:57.942729 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.942710 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-metrics-certs\") pod \"maas-controller-65bb9fc5b4-vt9r7\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:57.942823 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.942749 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-webhook-cert\") pod \"maas-controller-65bb9fc5b4-vt9r7\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:57.953301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:57.953279 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5f7\" (UniqueName: \"kubernetes.io/projected/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-kube-api-access-5x5f7\") pod \"maas-controller-65bb9fc5b4-vt9r7\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:58.114059 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:58.113993 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:58.389987 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:58.389966 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-vt9r7"] Sep 10 17:46:58.391922 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:46:58.391895 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0d96dd_3cd1_4ad1_8568_a9f71dc525b2.slice/crio-0fa4d5010d565b0b334464bb52c94a5d945f686393f0a9bd8ff6e13ff133121b WatchSource:0}: Error finding container 0fa4d5010d565b0b334464bb52c94a5d945f686393f0a9bd8ff6e13ff133121b: Status 404 returned error can't find the container with id 0fa4d5010d565b0b334464bb52c94a5d945f686393f0a9bd8ff6e13ff133121b Sep 10 17:46:59.206697 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:59.206605 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" event={"ID":"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2","Type":"ContainerStarted","Data":"5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1"} Sep 10 17:46:59.206697 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:59.206652 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" event={"ID":"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2","Type":"ContainerStarted","Data":"0fa4d5010d565b0b334464bb52c94a5d945f686393f0a9bd8ff6e13ff133121b"} Sep 10 17:46:59.207117 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:59.206737 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:46:59.234788 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:46:59.234743 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" podStartSLOduration=1.785526028 podStartE2EDuration="2.234728059s" totalImagesPullingTime="449.202031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:46:59.233323782 +0000 UTC m=+2018.331222142" watchObservedRunningTime="2026-09-10 17:46:59.234728059 +0000 UTC m=+2018.332626418" Sep 10 17:47:10.213577 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:10.213544 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:47:21.705452 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:21.705414 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-vt9r7"] Sep 10 17:47:21.705953 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:21.705751 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" podUID="6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2" containerName="manager" containerID="cri-o://5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1" gracePeriod=30 Sep 10 17:47:21.950440 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:21.950416 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:47:22.040116 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.040023 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-webhook-cert\") pod \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " Sep 10 17:47:22.040116 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.040117 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-metrics-certs\") pod \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " Sep 10 17:47:22.040342 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.040179 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x5f7\" (UniqueName: \"kubernetes.io/projected/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-kube-api-access-5x5f7\") pod \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\" (UID: \"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2\") " Sep 10 17:47:22.042070 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.042022 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2" (UID: "6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:47:22.042187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.042062 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2" (UID: "6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:47:22.042187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.042105 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-kube-api-access-5x5f7" (OuterVolumeSpecName: "kube-api-access-5x5f7") pod "6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2" (UID: "6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2"). InnerVolumeSpecName "kube-api-access-5x5f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:47:22.141341 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.141307 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-metrics-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:47:22.141341 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.141336 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x5f7\" (UniqueName: \"kubernetes.io/projected/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-kube-api-access-5x5f7\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:47:22.141341 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.141345 2557 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2-webhook-cert\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:47:22.316132 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.316021 2557 generic.go:356] "Generic (PLEG): container finished" podID="6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2" containerID="5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1" exitCode=0 Sep 10 17:47:22.316132 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.316074 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" event={"ID":"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2","Type":"ContainerDied","Data":"5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1"} Sep 10 17:47:22.316132 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.316094 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" Sep 10 17:47:22.316383 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.316134 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-vt9r7" event={"ID":"6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2","Type":"ContainerDied","Data":"0fa4d5010d565b0b334464bb52c94a5d945f686393f0a9bd8ff6e13ff133121b"} Sep 10 17:47:22.316383 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.316167 2557 scope.go:117] "RemoveContainer" containerID="5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1" Sep 10 17:47:22.325453 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.325423 2557 scope.go:117] "RemoveContainer" containerID="5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1" Sep 10 17:47:22.325755 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:47:22.325736 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1\": container with ID starting with 5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1 not found: ID does not exist" containerID="5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1" Sep 10 17:47:22.325824 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.325771 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1"} err="failed to get container status \"5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1\": rpc error: code = NotFound desc = could not find container \"5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1\": container with ID starting with 5407be86f205539489b883734c10015abcf3b267787ac09041fa185a08f8a5b1 not found: ID does not exist" Sep 10 17:47:22.337618 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.337593 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-vt9r7"] Sep 10 17:47:22.343076 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:22.343052 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-vt9r7"] Sep 10 17:47:23.131188 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.131152 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-4n448"] Sep 10 17:47:23.131583 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.131492 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2" containerName="manager" Sep 10 17:47:23.131583 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.131504 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2" containerName="manager" Sep 10 17:47:23.131583 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.131517 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" Sep 10 17:47:23.131583 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.131523 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" Sep 10 17:47:23.131583 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.131533 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" Sep 10 17:47:23.131583 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.131538 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f9e963-90ce-43c3-a35c-ce9e164117ae" containerName="istio-proxy" Sep 10 17:47:23.131798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.131590 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2" containerName="manager" Sep 10 17:47:23.131798 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.131600 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a49cb4-ec56-4581-9b0f-93c60d2ac6bf" containerName="istio-proxy" Sep 10 17:47:23.134136 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.134118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.136776 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.136750 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 10 17:47:23.136776 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.136760 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8xsv\"" Sep 10 17:47:23.136954 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.136780 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 10 17:47:23.136954 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.136765 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 10 17:47:23.145694 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.145671 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-4n448"] Sep 10 17:47:23.250781 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.250752 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50129cd8-fe2a-47aa-a3d0-223c970783a8-metrics-certs\") pod \"maas-controller-65bb9fc5b4-4n448\" (UID: \"50129cd8-fe2a-47aa-a3d0-223c970783a8\") " pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.250781 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.250783 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sstc\" (UniqueName: \"kubernetes.io/projected/50129cd8-fe2a-47aa-a3d0-223c970783a8-kube-api-access-7sstc\") pod \"maas-controller-65bb9fc5b4-4n448\" (UID: \"50129cd8-fe2a-47aa-a3d0-223c970783a8\") " pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.250993 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.250810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50129cd8-fe2a-47aa-a3d0-223c970783a8-webhook-cert\") pod \"maas-controller-65bb9fc5b4-4n448\" (UID: \"50129cd8-fe2a-47aa-a3d0-223c970783a8\") " pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.351414 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.351380 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50129cd8-fe2a-47aa-a3d0-223c970783a8-metrics-certs\") pod \"maas-controller-65bb9fc5b4-4n448\" (UID: \"50129cd8-fe2a-47aa-a3d0-223c970783a8\") " pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.351414 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.351414 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sstc\" (UniqueName: \"kubernetes.io/projected/50129cd8-fe2a-47aa-a3d0-223c970783a8-kube-api-access-7sstc\") pod \"maas-controller-65bb9fc5b4-4n448\" (UID: \"50129cd8-fe2a-47aa-a3d0-223c970783a8\") " pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.351632 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.351441 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50129cd8-fe2a-47aa-a3d0-223c970783a8-webhook-cert\") pod \"maas-controller-65bb9fc5b4-4n448\" (UID: \"50129cd8-fe2a-47aa-a3d0-223c970783a8\") " pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.353803 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.353771 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/50129cd8-fe2a-47aa-a3d0-223c970783a8-webhook-cert\") pod \"maas-controller-65bb9fc5b4-4n448\" (UID: \"50129cd8-fe2a-47aa-a3d0-223c970783a8\") " pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.353912 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.353834 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50129cd8-fe2a-47aa-a3d0-223c970783a8-metrics-certs\") pod \"maas-controller-65bb9fc5b4-4n448\" (UID: \"50129cd8-fe2a-47aa-a3d0-223c970783a8\") " pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.359165 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.359142 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sstc\" (UniqueName: \"kubernetes.io/projected/50129cd8-fe2a-47aa-a3d0-223c970783a8-kube-api-access-7sstc\") pod \"maas-controller-65bb9fc5b4-4n448\" (UID: \"50129cd8-fe2a-47aa-a3d0-223c970783a8\") " pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.385479 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.385419 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2" path="/var/lib/kubelet/pods/6e0d96dd-3cd1-4ad1-8568-a9f71dc525b2/volumes" Sep 10 17:47:23.443305 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.443280 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:23.716592 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:23.716559 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65bb9fc5b4-4n448"] Sep 10 17:47:23.719207 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:47:23.719178 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50129cd8_fe2a_47aa_a3d0_223c970783a8.slice/crio-4fa5b23e9713db0354b64208194bb18ef06c129f33bddc41040932ce9c5ec7a2 WatchSource:0}: Error finding container 4fa5b23e9713db0354b64208194bb18ef06c129f33bddc41040932ce9c5ec7a2: Status 404 returned error can't find the container with id 4fa5b23e9713db0354b64208194bb18ef06c129f33bddc41040932ce9c5ec7a2 Sep 10 17:47:24.327813 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:24.327729 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-4n448" event={"ID":"50129cd8-fe2a-47aa-a3d0-223c970783a8","Type":"ContainerStarted","Data":"92387e116d6cb22582620f882f1674baf43cdfd20db989caf57f4073baca3eb4"} Sep 10 17:47:24.327813 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:24.327779 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65bb9fc5b4-4n448" event={"ID":"50129cd8-fe2a-47aa-a3d0-223c970783a8","Type":"ContainerStarted","Data":"4fa5b23e9713db0354b64208194bb18ef06c129f33bddc41040932ce9c5ec7a2"} Sep 10 17:47:24.328232 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:24.327872 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:47:24.349155 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:24.349105 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65bb9fc5b4-4n448" podStartSLOduration=1.013568195 podStartE2EDuration="1.349088871s" totalImagesPullingTime="335.520676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:47:24.346765964 +0000 UTC m=+2043.444664324" watchObservedRunningTime="2026-09-10 17:47:24.349088871 +0000 UTC m=+2043.446987256" Sep 10 17:47:35.334176 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:47:35.334144 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65bb9fc5b4-4n448" Sep 10 17:48:21.450968 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:48:21.450853 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:48:21.457254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:48:21.457237 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n64hg_a570749c-795f-416e-9fb9-df195c4c80a2/ovn-acl-logging/0.log" Sep 10 17:50:20.807024 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.806992 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm"] Sep 10 17:50:20.810290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.810270 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:20.812882 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.812858 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-60869e21-kserve-self-signed-certs\"" Sep 10 17:50:20.827159 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.827134 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm"] Sep 10 17:50:20.912995 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.912947 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-dshm\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:20.913187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.913009 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-home\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:20.913187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.913109 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad32f65-2018-46bd-ad87-0646350130db-tls-certs\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:20.913187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.913127 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-tmp-dir\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:20.913187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.913151 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgt5r\" (UniqueName: \"kubernetes.io/projected/4ad32f65-2018-46bd-ad87-0646350130db-kube-api-access-pgt5r\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:20.913187 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:20.913173 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-model-cache\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.014285 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.014244 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-tmp-dir\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.014462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.014298 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgt5r\" (UniqueName: \"kubernetes.io/projected/4ad32f65-2018-46bd-ad87-0646350130db-kube-api-access-pgt5r\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.014462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.014334 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-model-cache\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.014462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.014377 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-dshm\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.014462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.014429 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-home\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.014647 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.014493 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad32f65-2018-46bd-ad87-0646350130db-tls-certs\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.014770 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.014731 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-tmp-dir\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.014770 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.014749 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-model-cache\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.014910 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.014889 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-home\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.016628 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.016606 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-dshm\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.016938 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.016922 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad32f65-2018-46bd-ad87-0646350130db-tls-certs\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.027155 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.027129 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgt5r\" (UniqueName: \"kubernetes.io/projected/4ad32f65-2018-46bd-ad87-0646350130db-kube-api-access-pgt5r\") pod \"e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.120223 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.120142 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:21.387671 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:21.387604 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm"] Sep 10 17:50:21.389820 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:50:21.389796 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ad32f65_2018_46bd_ad87_0646350130db.slice/crio-b8f48d30361b8d772a13224b2a23563c066e0305279b5023f6b439b172061c2a WatchSource:0}: Error finding container b8f48d30361b8d772a13224b2a23563c066e0305279b5023f6b439b172061c2a: Status 404 returned error can't find the container with id b8f48d30361b8d772a13224b2a23563c066e0305279b5023f6b439b172061c2a Sep 10 17:50:22.124545 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:22.124506 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" event={"ID":"4ad32f65-2018-46bd-ad87-0646350130db","Type":"ContainerStarted","Data":"2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075"} Sep 10 17:50:22.124545 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:22.124555 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" event={"ID":"4ad32f65-2018-46bd-ad87-0646350130db","Type":"ContainerStarted","Data":"b8f48d30361b8d772a13224b2a23563c066e0305279b5023f6b439b172061c2a"} Sep 10 17:50:22.142290 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:22.142246 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" podStartSLOduration=2.142233465 podStartE2EDuration="2.142233465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:50:22.140443481 +0000 UTC m=+2221.238341841" watchObservedRunningTime="2026-09-10 17:50:22.142233465 +0000 UTC m=+2221.240131824" Sep 10 17:50:31.120394 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:31.120341 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:31.120394 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:31.120409 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:31.132455 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:31.132432 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:31.182839 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:31.182813 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:50:57.238660 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:50:57.238573 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm"] Sep 10 17:51:12.258659 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.258620 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" podUID="4ad32f65-2018-46bd-ad87-0646350130db" containerName="main" containerID="cri-o://2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075" gracePeriod=45 Sep 10 17:51:12.495760 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.495735 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:51:12.660251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660168 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad32f65-2018-46bd-ad87-0646350130db-tls-certs\") pod \"4ad32f65-2018-46bd-ad87-0646350130db\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " Sep 10 17:51:12.660251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660214 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-dshm\") pod \"4ad32f65-2018-46bd-ad87-0646350130db\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " Sep 10 17:51:12.660251 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660242 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgt5r\" (UniqueName: \"kubernetes.io/projected/4ad32f65-2018-46bd-ad87-0646350130db-kube-api-access-pgt5r\") pod \"4ad32f65-2018-46bd-ad87-0646350130db\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " Sep 10 17:51:12.660475 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660296 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-model-cache\") pod \"4ad32f65-2018-46bd-ad87-0646350130db\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " Sep 10 17:51:12.660523 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660493 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-home\") pod \"4ad32f65-2018-46bd-ad87-0646350130db\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " Sep 10 17:51:12.660568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660530 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-tmp-dir\") pod \"4ad32f65-2018-46bd-ad87-0646350130db\" (UID: \"4ad32f65-2018-46bd-ad87-0646350130db\") " Sep 10 17:51:12.660568 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660558 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ad32f65-2018-46bd-ad87-0646350130db" (UID: "4ad32f65-2018-46bd-ad87-0646350130db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:51:12.660690 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660664 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-home" (OuterVolumeSpecName: "home") pod "4ad32f65-2018-46bd-ad87-0646350130db" (UID: "4ad32f65-2018-46bd-ad87-0646350130db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:51:12.660817 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660802 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-model-cache\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:12.660902 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660818 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-home\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:12.660902 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.660802 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ad32f65-2018-46bd-ad87-0646350130db" (UID: "4ad32f65-2018-46bd-ad87-0646350130db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:51:12.662502 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.662483 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-dshm" (OuterVolumeSpecName: "dshm") pod "4ad32f65-2018-46bd-ad87-0646350130db" (UID: "4ad32f65-2018-46bd-ad87-0646350130db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 17:51:12.662757 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.662741 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad32f65-2018-46bd-ad87-0646350130db-kube-api-access-pgt5r" (OuterVolumeSpecName: "kube-api-access-pgt5r") pod "4ad32f65-2018-46bd-ad87-0646350130db" (UID: "4ad32f65-2018-46bd-ad87-0646350130db"). InnerVolumeSpecName "kube-api-access-pgt5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:51:12.662827 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.662753 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad32f65-2018-46bd-ad87-0646350130db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ad32f65-2018-46bd-ad87-0646350130db" (UID: "4ad32f65-2018-46bd-ad87-0646350130db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:51:12.761485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.761444 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-tmp-dir\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:12.761485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.761478 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad32f65-2018-46bd-ad87-0646350130db-tls-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:12.761485 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.761489 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad32f65-2018-46bd-ad87-0646350130db-dshm\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:12.761723 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:12.761501 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgt5r\" (UniqueName: \"kubernetes.io/projected/4ad32f65-2018-46bd-ad87-0646350130db-kube-api-access-pgt5r\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:13.375522 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:13.375489 2557 generic.go:356] "Generic (PLEG): container finished" podID="4ad32f65-2018-46bd-ad87-0646350130db" containerID="2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075" exitCode=0 Sep 10 17:51:13.375927 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:13.375531 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" Sep 10 17:51:13.375927 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:13.375570 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" event={"ID":"4ad32f65-2018-46bd-ad87-0646350130db","Type":"ContainerDied","Data":"2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075"} Sep 10 17:51:13.375927 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:13.375613 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm" event={"ID":"4ad32f65-2018-46bd-ad87-0646350130db","Type":"ContainerDied","Data":"b8f48d30361b8d772a13224b2a23563c066e0305279b5023f6b439b172061c2a"} Sep 10 17:51:13.375927 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:13.375629 2557 scope.go:117] "RemoveContainer" containerID="2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075" Sep 10 17:51:13.385457 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:13.385443 2557 scope.go:117] "RemoveContainer" containerID="2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075" Sep 10 17:51:13.385740 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:51:13.385723 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075\": container with ID starting with 2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075 not found: ID does not exist" containerID="2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075" Sep 10 17:51:13.385805 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:13.385747 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075"} err="failed to get container status \"2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075\": rpc error: code = NotFound desc = could not find container \"2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075\": container with ID starting with 2ff1230c041e2fc15293c129732542a393f2c99344b924489fc3873198f83075 not found: ID does not exist" Sep 10 17:51:13.396988 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:13.396962 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm"] Sep 10 17:51:13.400245 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:13.400225 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-60869e21-kserve-66d68cf9cf-lz7qm"] Sep 10 17:51:15.387710 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:15.387676 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad32f65-2018-46bd-ad87-0646350130db" path="/var/lib/kubelet/pods/4ad32f65-2018-46bd-ad87-0646350130db/volumes" Sep 10 17:51:16.448928 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.448892 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt"] Sep 10 17:51:16.449314 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.449270 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad32f65-2018-46bd-ad87-0646350130db" containerName="main" Sep 10 17:51:16.449314 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.449281 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad32f65-2018-46bd-ad87-0646350130db" containerName="main" Sep 10 17:51:16.449388 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.449360 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad32f65-2018-46bd-ad87-0646350130db" containerName="main" Sep 10 17:51:16.453743 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.453727 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.467993 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.467968 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt"] Sep 10 17:51:16.593305 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.593274 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008e161f-ad35-4e04-ad9b-4f7bbdc12491-maas-api-tls\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.593305 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.593309 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/008e161f-ad35-4e04-ad9b-4f7bbdc12491-metrics-certs\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.593510 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.593354 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7ht\" (UniqueName: \"kubernetes.io/projected/008e161f-ad35-4e04-ad9b-4f7bbdc12491-kube-api-access-fr7ht\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.593510 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.593384 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008e161f-ad35-4e04-ad9b-4f7bbdc12491-openshift-service-ca\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.694587 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.694558 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7ht\" (UniqueName: \"kubernetes.io/projected/008e161f-ad35-4e04-ad9b-4f7bbdc12491-kube-api-access-fr7ht\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.694761 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.694590 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008e161f-ad35-4e04-ad9b-4f7bbdc12491-openshift-service-ca\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.694761 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.694646 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008e161f-ad35-4e04-ad9b-4f7bbdc12491-maas-api-tls\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.694761 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.694665 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/008e161f-ad35-4e04-ad9b-4f7bbdc12491-metrics-certs\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.695510 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.695485 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/008e161f-ad35-4e04-ad9b-4f7bbdc12491-openshift-service-ca\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.697479 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.697450 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/008e161f-ad35-4e04-ad9b-4f7bbdc12491-metrics-certs\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.697681 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.697661 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/008e161f-ad35-4e04-ad9b-4f7bbdc12491-maas-api-tls\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.702018 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.701966 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7ht\" (UniqueName: \"kubernetes.io/projected/008e161f-ad35-4e04-ad9b-4f7bbdc12491-kube-api-access-fr7ht\") pod \"maas-api-ddc7f4579-z68pt\" (UID: \"008e161f-ad35-4e04-ad9b-4f7bbdc12491\") " pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:16.765184 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:16.765144 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:17.057466 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:17.057267 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt"] Sep 10 17:51:17.063496 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:17.063468 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 17:51:17.397069 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:17.396963 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" event={"ID":"008e161f-ad35-4e04-ad9b-4f7bbdc12491","Type":"ContainerStarted","Data":"99067c35d974ed9a570caea5213e22386d231cddb7e047549b1c386c598d17b9"} Sep 10 17:51:18.402848 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:18.402810 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" event={"ID":"008e161f-ad35-4e04-ad9b-4f7bbdc12491","Type":"ContainerStarted","Data":"906d2c724616489b59df9090ad04965a0ed88cbeea48742de815a57152ce9f6b"} Sep 10 17:51:18.403266 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:18.402918 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:18.420508 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:18.420463 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" podStartSLOduration=2.024250295 podStartE2EDuration="2.420449498s" totalImagesPullingTime="396.199203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:51:18.417932323 +0000 UTC m=+2277.515830683" watchObservedRunningTime="2026-09-10 17:51:18.420449498 +0000 UTC m=+2277.518347857" Sep 10 17:51:21.696701 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:21.696669 2557 scope.go:117] "RemoveContainer" containerID="19e2d2eef95a6f209d09b08fefee7076668824cfb61b8fc10e660baaa71e7935" Sep 10 17:51:24.412591 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:24.412557 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-ddc7f4579-z68pt" Sep 10 17:51:32.589953 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:32.589916 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb"] Sep 10 17:51:32.590564 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:32.590195 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" podUID="e980a408-f795-463a-852d-5386cc740a1c" containerName="maas-api" containerID="cri-o://33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1" gracePeriod=30 Sep 10 17:51:33.831700 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:33.831677 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:51:33.940340 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:33.940249 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e980a408-f795-463a-852d-5386cc740a1c-openshift-service-ca\") pod \"e980a408-f795-463a-852d-5386cc740a1c\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " Sep 10 17:51:33.940340 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:33.940329 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-maas-api-tls\") pod \"e980a408-f795-463a-852d-5386cc740a1c\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " Sep 10 17:51:33.940567 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:33.940376 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-metrics-certs\") pod \"e980a408-f795-463a-852d-5386cc740a1c\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " Sep 10 17:51:33.940567 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:33.940404 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nds5h\" (UniqueName: \"kubernetes.io/projected/e980a408-f795-463a-852d-5386cc740a1c-kube-api-access-nds5h\") pod \"e980a408-f795-463a-852d-5386cc740a1c\" (UID: \"e980a408-f795-463a-852d-5386cc740a1c\") " Sep 10 17:51:33.940683 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:33.940559 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e980a408-f795-463a-852d-5386cc740a1c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e980a408-f795-463a-852d-5386cc740a1c" (UID: "e980a408-f795-463a-852d-5386cc740a1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 17:51:33.942366 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:33.942340 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e980a408-f795-463a-852d-5386cc740a1c" (UID: "e980a408-f795-463a-852d-5386cc740a1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:51:33.942462 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:33.942424 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e980a408-f795-463a-852d-5386cc740a1c-kube-api-access-nds5h" (OuterVolumeSpecName: "kube-api-access-nds5h") pod "e980a408-f795-463a-852d-5386cc740a1c" (UID: "e980a408-f795-463a-852d-5386cc740a1c"). InnerVolumeSpecName "kube-api-access-nds5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 17:51:33.942504 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:33.942489 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e980a408-f795-463a-852d-5386cc740a1c" (UID: "e980a408-f795-463a-852d-5386cc740a1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 17:51:34.041661 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.041627 2557 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-metrics-certs\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:34.041661 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.041659 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nds5h\" (UniqueName: \"kubernetes.io/projected/e980a408-f795-463a-852d-5386cc740a1c-kube-api-access-nds5h\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:34.041661 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.041668 2557 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e980a408-f795-463a-852d-5386cc740a1c-openshift-service-ca\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:34.041870 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.041676 2557 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e980a408-f795-463a-852d-5386cc740a1c-maas-api-tls\") on node \"ip-10-0-133-191.ec2.internal\" DevicePath \"\"" Sep 10 17:51:34.474464 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.474431 2557 generic.go:356] "Generic (PLEG): container finished" podID="e980a408-f795-463a-852d-5386cc740a1c" containerID="33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1" exitCode=0 Sep 10 17:51:34.474626 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.474471 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" Sep 10 17:51:34.474626 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.474511 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" event={"ID":"e980a408-f795-463a-852d-5386cc740a1c","Type":"ContainerDied","Data":"33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1"} Sep 10 17:51:34.474626 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.474564 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb" event={"ID":"e980a408-f795-463a-852d-5386cc740a1c","Type":"ContainerDied","Data":"72d41c68ac133adfc4298605211275b0869836608bd2cd37d2e01ca6126c2f90"} Sep 10 17:51:34.474626 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.474591 2557 scope.go:117] "RemoveContainer" containerID="33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1" Sep 10 17:51:34.484947 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.484929 2557 scope.go:117] "RemoveContainer" containerID="33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1" Sep 10 17:51:34.485225 ip-10-0-133-191 kubenswrapper[2557]: E0910 17:51:34.485205 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1\": container with ID starting with 33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1 not found: ID does not exist" containerID="33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1" Sep 10 17:51:34.485284 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.485249 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1"} err="failed to get container status \"33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1\": rpc error: code = NotFound desc = could not find container \"33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1\": container with ID starting with 33ce5caa4c012cb21dda6f190d0942ac89eaaf99bb3438436521aaf121b28ba1 not found: ID does not exist" Sep 10 17:51:34.496011 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.495991 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb"] Sep 10 17:51:34.501324 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:34.501302 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c68568df5-xrkdb"] Sep 10 17:51:35.386581 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:35.386546 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e980a408-f795-463a-852d-5386cc740a1c" path="/var/lib/kubelet/pods/e980a408-f795-463a-852d-5386cc740a1c/volumes" Sep 10 17:51:55.238104 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:55.238072 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gctlv_0847ed17-5118-45dc-b679-81af2c15a17c/manager/0.log" Sep 10 17:51:55.353186 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:55.353155 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z2hkv_ca796d01-4160-41df-a46b-20fdfea7324a/manager/0.log" Sep 10 17:51:55.479236 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:55.479207 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65bb9fc5b4-4n448_50129cd8-fe2a-47aa-a3d0-223c970783a8/manager/0.log" Sep 10 17:51:55.960367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:55.960338 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dc495f6d-twbtn_ccc1e89d-594c-41fd-a333-8a1ec6c061f8/copy-manifests/0.log" Sep 10 17:51:55.967397 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:55.967371 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dc495f6d-twbtn_ccc1e89d-594c-41fd-a333-8a1ec6c061f8/manager/0.log" Sep 10 17:51:56.547692 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:56.547659 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-ddc7f4579-z68pt_008e161f-ad35-4e04-ad9b-4f7bbdc12491/maas-api/0.log" Sep 10 17:51:56.776753 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:56.776721 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817705-g2xzr_e01138b4-1007-43a3-af9d-cffaf01a359f/cleanup/0.log" Sep 10 17:51:56.889367 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:56.889292 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ws6w9_455b8f60-aa7a-465e-8177-51b794eddc94/postgres/0.log" Sep 10 17:51:57.924885 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:57.924861 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-g9qvc_a5d4e00e-0ab2-4aa2-8201-9db87b49f957/manager/0.log" Sep 10 17:51:58.882277 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:58.882239 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-thzdx_6a79dedd-0eae-4a3c-a898-5d1c0b331859/discovery/0.log" Sep 10 17:51:59.113935 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:59.113904 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555bd9596f-kmf2j_bc7f6dae-5899-4d92-bbaa-902562678414/kube-auth-proxy/0.log" Sep 10 17:51:59.236922 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:59.236847 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8k8f9_0578c7d9-e911-4d2d-9266-7862658d07f4/istio-proxy/1.log" Sep 10 17:51:59.355564 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:59.355534 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lp52j_b251030e-8917-4c97-a641-542c0dc76045/payload-pre-processing/0.log" Sep 10 17:51:59.476112 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:59.476081 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w7ck4_c21f986f-64b2-4347-af53-2d38959e718f/payload-processing/0.log" Sep 10 17:51:59.593154 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:59.593124 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bf89c887b-crb9s_52ef40d5-e77b-4b0d-acdd-cebf3147dcb4/router/0.log" Sep 10 17:51:59.970622 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:51:59.970547 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-2tq9f_f2b73d20-8756-487d-a6ee-8892bc1fa561/main/0.log" Sep 10 17:52:00.088055 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:00.088003 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-t86ws_8da6b695-6834-4768-87f2-d5a29a900678/main/0.log" Sep 10 17:52:09.477588 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:09.477550 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mrzvh_d2370b0a-f5f3-4ffb-a627-7a6cceae329e/global-pull-secret-syncer/0.log" Sep 10 17:52:09.619558 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:09.619531 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p25s2_c4010ee5-e8c9-4866-be3d-8d7b3312e4fc/konnectivity-agent/0.log" Sep 10 17:52:09.673300 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:09.673272 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-191.ec2.internal_af1fd4d122936dd15efa29d3d763213a/haproxy/0.log" Sep 10 17:52:15.176612 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:15.176581 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-g9qvc_a5d4e00e-0ab2-4aa2-8201-9db87b49f957/manager/0.log" Sep 10 17:52:17.209695 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:17.209592 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-jbpqj_0067abc0-3ae9-4e8e-b736-e0a48a9fd797/cluster-monitoring-operator/0.log" Sep 10 17:52:17.571561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:17.571530 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8qpv_61f7efb5-13ae-4b0a-ad41-043d0c440ce4/node-exporter/0.log" Sep 10 17:52:17.594475 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:17.594448 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8qpv_61f7efb5-13ae-4b0a-ad41-043d0c440ce4/kube-rbac-proxy/0.log" Sep 10 17:52:17.617158 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:17.617131 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8qpv_61f7efb5-13ae-4b0a-ad41-043d0c440ce4/init-textfile/0.log" Sep 10 17:52:20.140156 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.140118 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79"] Sep 10 17:52:20.140528 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.140495 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e980a408-f795-463a-852d-5386cc740a1c" containerName="maas-api" Sep 10 17:52:20.140528 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.140508 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="e980a408-f795-463a-852d-5386cc740a1c" containerName="maas-api" Sep 10 17:52:20.140602 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.140567 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="e980a408-f795-463a-852d-5386cc740a1c" containerName="maas-api" Sep 10 17:52:20.143683 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.143663 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.147608 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.147586 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7ws5r\"/\"kube-root-ca.crt\"" Sep 10 17:52:20.147707 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.147586 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7ws5r\"/\"openshift-service-ca.crt\"" Sep 10 17:52:20.148193 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.148178 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7ws5r\"/\"default-dockercfg-9c22d\"" Sep 10 17:52:20.157894 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.157866 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79"] Sep 10 17:52:20.240073 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.240015 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-proc\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.240266 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.240080 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htpw\" (UniqueName: \"kubernetes.io/projected/a514cc15-39af-48c0-a638-532a0580b9d1-kube-api-access-6htpw\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.240266 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.240109 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-sys\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.240266 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.240244 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-lib-modules\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.240414 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.240277 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-podres\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.341561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.341526 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-lib-modules\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.341561 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.341559 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-podres\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.341804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.341588 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-proc\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.341804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.341610 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htpw\" (UniqueName: \"kubernetes.io/projected/a514cc15-39af-48c0-a638-532a0580b9d1-kube-api-access-6htpw\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.341804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.341634 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-sys\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.341804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.341723 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-podres\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.341804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.341734 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-proc\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.341804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.341741 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-sys\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.341804 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.341723 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a514cc15-39af-48c0-a638-532a0580b9d1-lib-modules\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.349987 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.349959 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htpw\" (UniqueName: \"kubernetes.io/projected/a514cc15-39af-48c0-a638-532a0580b9d1-kube-api-access-6htpw\") pod \"perf-node-gather-daemonset-x4z79\" (UID: \"a514cc15-39af-48c0-a638-532a0580b9d1\") " pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.452949 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.452866 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:20.725937 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:20.725910 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79"] Sep 10 17:52:20.728428 ip-10-0-133-191 kubenswrapper[2557]: W0910 17:52:20.728401 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda514cc15_39af_48c0_a638_532a0580b9d1.slice/crio-235e24d7c2d2a4c6cf59d079cbef64488f901bd50da682b2f0231f2fc7e96a33 WatchSource:0}: Error finding container 235e24d7c2d2a4c6cf59d079cbef64488f901bd50da682b2f0231f2fc7e96a33: Status 404 returned error can't find the container with id 235e24d7c2d2a4c6cf59d079cbef64488f901bd50da682b2f0231f2fc7e96a33 Sep 10 17:52:21.692758 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:21.692714 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" event={"ID":"a514cc15-39af-48c0-a638-532a0580b9d1","Type":"ContainerStarted","Data":"e4a6ea4be2e0f0950dd153f7509174062839d453c1e57a8e6764187458c00dee"} Sep 10 17:52:21.693190 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:21.692781 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" event={"ID":"a514cc15-39af-48c0-a638-532a0580b9d1","Type":"ContainerStarted","Data":"235e24d7c2d2a4c6cf59d079cbef64488f901bd50da682b2f0231f2fc7e96a33"} Sep 10 17:52:21.693190 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:21.692855 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:21.708276 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:21.708231 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" podStartSLOduration=1.708214158 podStartE2EDuration="1.708214158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:52:21.707935173 +0000 UTC m=+2340.805833533" watchObservedRunningTime="2026-09-10 17:52:21.708214158 +0000 UTC m=+2340.806112577" Sep 10 17:52:22.412992 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:22.412965 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lsk5m_c81cc54e-dfe7-45a1-a04c-b9abba50450d/dns/0.log" Sep 10 17:52:22.434845 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:22.434812 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lsk5m_c81cc54e-dfe7-45a1-a04c-b9abba50450d/kube-rbac-proxy/0.log" Sep 10 17:52:22.574969 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:22.574935 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x275d_6e5ea32f-27b0-4267-98b1-9ceb809f9f44/dns-node-resolver/0.log" Sep 10 17:52:23.140280 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:23.140252 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zdrnp_a38b50d7-4bd2-4605-8896-fd0d0f5b35d8/node-ca/0.log" Sep 10 17:52:24.502001 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:24.501960 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-thzdx_6a79dedd-0eae-4a3c-a898-5d1c0b331859/discovery/0.log" Sep 10 17:52:24.549639 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:24.549606 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555bd9596f-kmf2j_bc7f6dae-5899-4d92-bbaa-902562678414/kube-auth-proxy/0.log" Sep 10 17:52:24.582576 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:24.582548 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8k8f9_0578c7d9-e911-4d2d-9266-7862658d07f4/istio-proxy/0.log" Sep 10 17:52:24.591252 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:24.591227 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8k8f9_0578c7d9-e911-4d2d-9266-7862658d07f4/istio-proxy/1.log" Sep 10 17:52:24.626415 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:24.626391 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lp52j_b251030e-8917-4c97-a641-542c0dc76045/payload-pre-processing/0.log" Sep 10 17:52:24.662020 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:24.661992 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w7ck4_c21f986f-64b2-4347-af53-2d38959e718f/payload-processing/0.log" Sep 10 17:52:24.683308 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:24.683288 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bf89c887b-crb9s_52ef40d5-e77b-4b0d-acdd-cebf3147dcb4/router/0.log" Sep 10 17:52:25.582870 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:25.582846 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9549r_e8bd1e76-8725-4094-a2f5-b4d94d75abf7/serve-healthcheck-canary/0.log" Sep 10 17:52:26.296210 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:26.296184 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qz6j_d07b5b9b-b239-4200-8984-753fe31b0551/kube-rbac-proxy/0.log" Sep 10 17:52:26.317958 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:26.317934 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qz6j_d07b5b9b-b239-4200-8984-753fe31b0551/exporter/0.log" Sep 10 17:52:26.340803 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:26.340778 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6qz6j_d07b5b9b-b239-4200-8984-753fe31b0551/extractor/0.log" Sep 10 17:52:27.706921 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:27.706894 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7ws5r/perf-node-gather-daemonset-x4z79" Sep 10 17:52:28.547048 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:28.547009 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gctlv_0847ed17-5118-45dc-b679-81af2c15a17c/manager/0.log" Sep 10 17:52:28.725105 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:28.725070 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z2hkv_ca796d01-4160-41df-a46b-20fdfea7324a/manager/0.log" Sep 10 17:52:28.770301 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:28.770267 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-65bb9fc5b4-4n448_50129cd8-fe2a-47aa-a3d0-223c970783a8/manager/0.log" Sep 10 17:52:29.103546 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:29.103517 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dc495f6d-twbtn_ccc1e89d-594c-41fd-a333-8a1ec6c061f8/manager/0.log" Sep 10 17:52:29.123502 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:29.123476 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dc495f6d-twbtn_ccc1e89d-594c-41fd-a333-8a1ec6c061f8/copy-manifests/0.log" Sep 10 17:52:30.555363 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:30.555337 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-q8428_5d48625f-bf7e-45ba-a9cd-8e42697bc0ab/openshift-lws-operator/0.log" Sep 10 17:52:31.508124 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:31.508026 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-thzdx_6a79dedd-0eae-4a3c-a898-5d1c0b331859/discovery/0.log" Sep 10 17:52:31.534539 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:31.534513 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555bd9596f-kmf2j_bc7f6dae-5899-4d92-bbaa-902562678414/kube-auth-proxy/0.log" Sep 10 17:52:31.553463 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:31.553428 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8k8f9_0578c7d9-e911-4d2d-9266-7862658d07f4/istio-proxy/0.log" Sep 10 17:52:31.563471 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:31.563448 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8k8f9_0578c7d9-e911-4d2d-9266-7862658d07f4/istio-proxy/1.log" Sep 10 17:52:31.589573 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:31.589550 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lp52j_b251030e-8917-4c97-a641-542c0dc76045/payload-pre-processing/0.log" Sep 10 17:52:31.613551 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:31.613525 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w7ck4_c21f986f-64b2-4347-af53-2d38959e718f/payload-processing/0.log" Sep 10 17:52:31.626553 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:31.626538 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bf89c887b-crb9s_52ef40d5-e77b-4b0d-acdd-cebf3147dcb4/router/0.log" Sep 10 17:52:33.441322 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:33.441287 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-thzdx_6a79dedd-0eae-4a3c-a898-5d1c0b331859/discovery/0.log" Sep 10 17:52:33.477233 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:33.477204 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-555bd9596f-kmf2j_bc7f6dae-5899-4d92-bbaa-902562678414/kube-auth-proxy/0.log" Sep 10 17:52:33.499120 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:33.499081 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8k8f9_0578c7d9-e911-4d2d-9266-7862658d07f4/istio-proxy/0.log" Sep 10 17:52:33.507869 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:33.507846 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-8k8f9_0578c7d9-e911-4d2d-9266-7862658d07f4/istio-proxy/1.log" Sep 10 17:52:33.538138 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:33.538114 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lp52j_b251030e-8917-4c97-a641-542c0dc76045/payload-pre-processing/0.log" Sep 10 17:52:33.566759 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:33.566731 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w7ck4_c21f986f-64b2-4347-af53-2d38959e718f/payload-processing/0.log" Sep 10 17:52:33.588672 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:33.588648 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-bf89c887b-crb9s_52ef40d5-e77b-4b0d-acdd-cebf3147dcb4/router/0.log" Sep 10 17:52:35.946119 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:35.946086 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-2tz48_15c87953-16b7-4c34-a7b0-eb3b79b09fa2/migrator/0.log" Sep 10 17:52:35.965380 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:35.965352 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-2tz48_15c87953-16b7-4c34-a7b0-eb3b79b09fa2/graceful-termination/0.log" Sep 10 17:52:36.304918 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:36.304887 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-bdf5l_b96006c6-5198-45c0-ad2f-aa8a703934e6/kube-storage-version-migrator-operator/1.log" Sep 10 17:52:36.305875 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:36.305856 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-bdf5l_b96006c6-5198-45c0-ad2f-aa8a703934e6/kube-storage-version-migrator-operator/0.log" Sep 10 17:52:37.330302 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:37.330265 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6qtnb_62a0c4e9-0ee9-466e-adc0-335dacce4f3b/kube-multus/0.log" Sep 10 17:52:37.620378 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:37.620351 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljl7_6b7ddc91-a214-420a-8108-bd6d6ee44e88/kube-multus-additional-cni-plugins/0.log" Sep 10 17:52:37.657348 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:37.657298 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljl7_6b7ddc91-a214-420a-8108-bd6d6ee44e88/egress-router-binary-copy/0.log" Sep 10 17:52:37.699218 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:37.699194 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljl7_6b7ddc91-a214-420a-8108-bd6d6ee44e88/cni-plugins/0.log" Sep 10 17:52:37.737312 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:37.737283 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljl7_6b7ddc91-a214-420a-8108-bd6d6ee44e88/bond-cni-plugin/0.log" Sep 10 17:52:37.775230 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:37.775204 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljl7_6b7ddc91-a214-420a-8108-bd6d6ee44e88/routeoverride-cni/0.log" Sep 10 17:52:37.821449 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:37.821425 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljl7_6b7ddc91-a214-420a-8108-bd6d6ee44e88/whereabouts-cni-bincopy/0.log" Sep 10 17:52:37.857254 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:37.857226 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rljl7_6b7ddc91-a214-420a-8108-bd6d6ee44e88/whereabouts-cni/0.log" Sep 10 17:52:38.454146 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:38.454119 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tgzqt_cb911006-5fc7-45f2-b42f-b44ccb6d0ac8/network-metrics-daemon/0.log" Sep 10 17:52:38.505312 ip-10-0-133-191 kubenswrapper[2557]: I0910 17:52:38.505285 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tgzqt_cb911006-5fc7-45f2-b42f-b44ccb6d0ac8/kube-rbac-proxy/0.log"